Coded Bias: How AI Is Learning to Think Like Us (and Why That’s a Problem)

For decades, we’ve held a shimmering, sci-fi vision of the future. It’s a world run by artificial intelligence, a world of sleek, impartial algorithms making decisions with a speed and rationality that we fragile humans could only dream of. We imagined AI as the ultimate antidote to our flawed, emotional, and biased minds. These systems, built on the pure logic of mathematics, would surely be free from the petty prejudices and mental glitches that have plagued humanity for millennia. They would hire the best candidate, approve the most deserving loan, and deliver the most objective justice.

This utopian vision is proving to be a profound and dangerous fantasy. As we delegate more and more of our critical decisions to automated systems, we are not escaping our biases. We are merely outsourcing them. We are teaching our own flawed patterns to machines, then hiding that bias behind a veneer of technological neutrality. We are taking the ghosts of our past prejudices and programming them into the architecture of our future.

This is the new frontier of cognitive bias: coded bias. It’s the digital manifestation of our oldest psychological failings. It’s a world where an algorithm, with no malice or intent, can learn to be racist, sexist, and exclusionary because it has been trained on the messy, biased data of human history. To navigate the coming decades, we must move past our awe of AI’s capabilities and develop a healthy, critical understanding of its limitations. We must become literate in the psychology of our own machines.

The Ghost in the Machine: How AI Learns Our Stereotypes

The promise of using AI in hiring seems so alluring. An algorithm could sift through thousands of resumes, immune to the human biases that have historically held back women and minorities, and select the most qualified candidates based on pure merit. What could possibly go wrong?

In 2018, the world found out. It was revealed that Amazon had been secretly working on an AI recruiting tool. The system was fed a decade’s worth of resumes submitted to the company, and its job was to learn the characteristics of a successful hire. But the AI didn’t learn about skills. It learned about history. Because the tech industry has been historically male-dominated, the historical data was overwhelmingly male.

The algorithm quickly taught itself that male candidates were preferable. It learned to penalize resumes that contained the word “women’s,” as in “women’s chess club captain.” It reportedly even downgraded graduates of two all-women’s colleges. Amazon’s engineers tried to edit the system to make it neutral, but they couldn’t guarantee it wouldn’t find new, more subtle ways to discriminate. They ultimately scrapped the entire project.

This is a perfect, if terrifying, example of how Stereotyping gets coded into AI. The AI itself isn’t “sexist.” It’s a pattern-matching machine. It ingested a dataset that reflected a biased reality, identified a pattern (maleness correlates with being hired), and logically concluded that this was a desirable trait. The algorithm didn’t learn our values; it learned our past behavior, warts and all.

The Problem of Proxies

The real danger lies in how algorithms use proxies. A proxy is an indirect measure of a desired outcome. An AI might be forbidden from using race as a factor in loan decisions. But it can still use proxies for race, like zip codes, which are often highly segregated. The AI might learn from historical data that people from certain zip codes have a higher default rate. Without any understanding of the systemic factors that create this disparity (like historical redlining and lack of access to banking), the algorithm will simply conclude that living in that zip code is a sign of being a high-risk borrower. It has effectively laundered a historical, societal bias through a seemingly neutral data point, creating a high-tech version of redlining.

This is the insidious nature of coded bias. It creates a feedback loop. The biased data produces a biased algorithm, which in turn makes biased decisions that create even more biased data for the next generation of algorithms to learn from.

The Algorithmic Echo Chamber: Supercharging the Availability Heuristic

We’ve already explored how our brains fall victim to the Availability Heuristic, our tendency to judge the likelihood of something by how easily examples of it come to mind. This is why we fear plane crashes more than car accidents; the dramatic, vivid images of a plane crash are far more “available” in our memory.

Now, consider the recommendation engines that govern our digital lives. Your YouTube homepage, your Netflix queue, your Spotify “Discover Weekly” playlist, and your news feed are all designed with one primary goal: to keep you engaged by showing you things you are likely to click on. To do this, they learn from your past behavior.

If you watch one video about a bizarre conspiracy theory out of idle curiosity, the algorithm notes this. It concludes, “Ah, this user is interested in conspiracy theories.” It then serves you another, slightly more extreme video on the topic. If you watch that one, it doubles down. Before you know it, your entire recommended list is a firehose of conspiratorial content.

The algorithm has created a powerful feedback loop for the Availability Heuristic. It takes a fleeting interest and, by making similar content constantly and readily available, it transforms that interest into what feels like a significant, pervasive, and important topic. It creates the illusion that “everyone is talking about this,” when in reality, you are in a personalized media bubble of one, curated by a machine that is simply trying to guess what you’ll click on next.

This isn’t just about entertainment. This same mechanism can shape our perception of political reality, our health fears, and our view of societal risk. The algorithm makes certain ideas, fears, and narratives hyper-available, distorting our mental map of the world in a way that is both incredibly subtle and profoundly impactful.

Automation Bias: Our Dangerous Trust in the Machine

Compounding this problem is our innate Automation Bias. This is our tendency to trust the suggestions and decisions made by automated systems, sometimes more than our own judgment. We see the output of a computer and assume it must be objective and correct.

When a GPS tells us to turn down a deserted-looking road, we often obey without question. When a recommendation engine suggests a video, we are more likely to see it as a credible suggestion than if a random stranger had recommended it. We grant the machine an unearned aura of authority.

This means we are less likely to be critical of the information presented to us by an algorithm. We let our guard down, assuming the system is neutral. This makes the Availability Heuristic feedback loop even more powerful. We’re not just being shown a distorted view of the world; we’re predisposed to believe it.

The Unseen Architecture: How Bias Hides in the Code Itself

So far, we’ve focused on how biased data creates biased outcomes. But bias can also be baked into the very design and architecture of the algorithm itself. The choices made by the human programmers—what data to include, what variables to prioritize, how to define “success”—are all laden with their own values and inherent biases.

Consider an algorithm designed to predict criminal recidivism, which is used in some court systems to help determine sentencing. How do you define “success” for this algorithm? Is a successful outcome simply “this person was not re-arrested”? This seems logical, but it hides a massive bias.

In the United States, policing is not uniform. Certain neighborhoods, often predominantly minority communities, are policed much more heavily than others. A person living in a heavily policed area is statistically more likely to be arrested for a minor infraction (like loitering or marijuana possession) than a person in a less-policed suburban neighborhood who is committing the same infraction.

Therefore, an algorithm trained on arrest data will logically conclude that people from that heavily policed neighborhood are at a higher risk of “recidivism.” The definition of success—”avoiding re-arrest”—is itself biased by real-world policing practices. The algorithm isn’t just reflecting a biased world; its very objective function is codifying and reinforcing that bias, creating a veneer of objective, data-driven justification for a deeply inequitable cycle.

The Myth of Neutrality

This reveals the central myth of our technological age: the myth that technology is neutral. Every technological tool is a product of human choices, and every human choice is subject to bias. An algorithm is an opinion, embedded in code. It is a set of priorities and definitions, written in the language of mathematics.

There is no “raw data.” Data is collected by people, cleaned by people, and interpreted by people (or by systems designed by people). The decision of what data to even collect is a human one. Are we collecting data on income but not on generational wealth? Are we collecting data on prior convictions but not on access to community support systems? Every choice shapes the “reality” the algorithm gets to see.

The Way Forward: Towards Algorithmic Accountability

The picture seems bleak, but it is not hopeless. The rise of coded bias is not an unstoppable force of nature; it is a design problem. And because it is a design problem, it has design solutions. The path forward requires a new kind of vigilance, a new kind of literacy, and a new commitment to accountability.

A Call for Transparency and Auditing

First, we must demand transparency. When a decision that significantly affects a person’s life (like a loan, a job application, or a parole hearing) is made with the help of an algorithm, we have a right to know. The “black box” approach, where companies treat their algorithms as untouchable trade secrets, is no longer acceptable.

Second, we need robust, independent auditing. Just as financial auditors scrutinize a company’s books, algorithmic auditors must be empowered to scrutinize a system’s code, data, and outcomes. These auditors should be diverse, multi-disciplinary teams of computer scientists, social scientists, and ethicists who can probe for coded biases and test for discriminatory outcomes.

Diversifying the Creators

The homogeneity of the tech industry is a major source of coded bias. A team of programmers who are all from similar demographic and socioeconomic backgrounds will have a limited, shared set of life experiences and, therefore, a shared set of blind spots. They are less likely to anticipate how their technology might negatively impact communities they are not a part of. Actively building more diverse and inclusive teams is not just a matter of social justice; it is a prerequisite for building safer and more equitable technology. A wider range of perspectives in the design phase will lead to a more robust and less biased product.

The age of AI is upon us, and with it comes a new and powerful vector for our oldest human failings. Our biases have escaped the confines of our skulls and now live in the cloud, operating at a scale and speed we can barely comprehend. We cannot afford to be naive about these new machines. We must approach them not with blind faith, but with critical wisdom, recognizing that every algorithm is a mirror. And if we don’t like the reflection we see, it is up to us, not the machine, to change.

MagTalk Discussion

Focus on Language

Vocabulary and Speaking

Alright, let’s get our magnifying glass out and look at the words that powered that article. When you’re talking about something as complex and futuristic as AI and bias, your vocabulary needs to be both precise and evocative. It has to make abstract ideas feel concrete. Let’s explore ten of the key terms we used and really unpack how you can use them to make your own language more impactful.

We’ll start with impartial. In the opening, we talked about the dream of “impartial algorithms.” To be impartial means to be neutral, fair, and not supporting any of the sides involved in an argument. A good judge is impartial. A good journalist strives for impartial reporting. It’s a cornerstone of justice and objectivity. By using it to describe our dream for AI, we’re highlighting our hope that machines could achieve a level of fairness that humans often struggle with. It’s a more formal and specific word than just “fair.”

Next up, veneer. We said we are hiding bias behind a “veneer of technological neutrality.” A veneer is a thin decorative covering of fine wood applied to a coarser wood or other material. Think of a cheap piece of furniture with a thin layer of expensive-looking wood on top. Metaphorically, a veneer is a deceptive or attractive outward show that conceals something of lesser quality underneath. Someone might have a veneer of confidence that hides deep insecurity. It’s the perfect word here because it suggests that the “objectivity” of AI is just a thin, superficial layer hiding the same old ugly biases underneath.

Let’s look at the word manifestation. We called coded bias the “digital manifestation of our oldest psychological failings.” A manifestation is an event, action, or object that clearly shows or embodies something, especially an abstract idea. A rash can be a manifestation of an allergy. A protest can be a manifestation of public anger. It means taking an idea or a feeling and making it visible and real. So, coded bias is our invisible psychological quirks made visible and real in the digital world.

Here’s a great adjective: alluring. We described the promise of using AI in hiring as “alluring.” Alluring means powerfully and mysteriously attractive or fascinating; seductive. It’s a stronger and more romantic word than just “attractive.” An alluring smile, an alluring offer. It suggests a pull that is almost irresistible, which captures the seductive promise of a perfectly fair and efficient hiring machine.

Let’s talk about the word scrapped. In the Amazon story, we said they ultimately “scrapped the entire project.” To scrap something is to discard or abandon it as no longer useful or viable. You might scrap an old car for its parts, or scrap a bad idea after realizing it won’t work. It’s a very decisive and final-sounding verb. It’s more informal and forceful than “canceled” or “discontinued.” It implies that the thing was broken beyond repair and had to be thrown away.

Now for insidious. We called coded bias “insidious in nature.” Insidious means proceeding in a gradual, subtle way, but with very harmful effects. It’s a close cousin of “pernicious,” but insidious often carries more of a sense of treachery or deceit. An insidious disease is one that spreads slowly without symptoms until it’s too late. An insidious rumor can quietly destroy a reputation. It’s a perfect word for coded bias because the harm is hidden, subtle, and spreads quietly through systems until its damage is widespread.

Let’s look at compounding. We said Automation Bias is “compounding this problem.” To compound a problem or a difficult situation is to make it worse by adding to it. Pouring water on a grease fire will compound the problem. Lying to cover up a mistake will only compound your difficulties. It’s a great verb that means more than just “to add to”; it specifically means to make a bad situation even worse.

Then there is recidivism. We talked about an algorithm designed to predict criminal “recidivism.” Recidivism is a technical term for the tendency of a convicted criminal to reoffend. A high recidivism rate means that many people who are released from prison go on to commit more crimes. While it’s a specific term from criminology, it’s a good word to know as it appears frequently in discussions about justice reform.

Here’s another great one: homogeneity. We mentioned the “homogeneity of the tech industry” as a source of bias. Homogeneity is the quality or state of being all the same or all of the same kind. It’s the opposite of diversity or heterogeneity. You might talk about the cultural homogeneity of a small town. The word itself is neutral, but in the context of teams or ecosystems, it’s often used to point out a lack of diversity as a weakness.

Finally, let’s discuss vector. We called AI a “new and powerful vector for our oldest human failings.” In mathematics and physics, a vector is a quantity that has both magnitude and direction. In medicine, a vector is an organism, like a mosquito, that transmits a disease from one animal or plant to another. Metaphorically, a vector is a means by which something is transmitted or carried. Social media can be a vector for misinformation. The new policy was a vector for change. It’s a very precise, scientific-sounding word for a carrier or a mode of transmission.

So, there we are: impartial, veneer, manifestation, alluring, scrapped, insidious, compounding, recidivism, homogeneity, and vector. Ten very potent words for your intellectual toolkit.

Now for our speaking skill. Today, let’s focus on the skill of explaining a causal chain. This is about showing how one thing leads to another, which leads to another. The article did this constantly. For example: “The biased data (A) produces a biased algorithm (B), which in turn makes biased decisions (C) that create even more biased data for the next generation (D).” Explaining these cause-and-effect relationships clearly is crucial for making a complex argument understandable.

The key is to use clear transition and signal words. Phrases like “This leads to…,” “As a result…,” “The consequence is…,” “This, in turn, causes…,” and “Therefore…” are the glue that holds a causal chain together. Without them, you just have a list of facts. With them, you have a logical narrative.

Here is your challenge: Pick a complex problem that interests you. It could be anything from climate change, to economic inflation, to why your favorite sports team is losing. Your task is to explain the problem by outlining a causal chain with at least three steps. Record yourself explaining it. For example: “The lack of rain (A) has led to a poor harvest. As a result (B), food prices are rising. The consequence of this is (C) that many families are struggling to afford groceries.” Listen back to your recording. Was the chain of logic clear? Did you use signal words to guide your listener from one step to the next? This skill is fundamental to persuasive and analytical speaking.

Grammar and Writing

Welcome to the writer’s workshop, where we translate complex ideas into powerful written communication. Today’s challenge is about looking to the future and crafting a persuasive argument about the intersection of technology and society.

The Writing Challenge:

Write a short, formal letter (around 500-750 words) to a public official, a CEO of a tech company, or the editor of a major newspaper. Your letter should express concern about the potential societal impact of a specific application of AI or algorithmic decision-making (e.g., in hiring, criminal justice, loan applications, or content moderation).

Your letter must:

1. Establish Credibility and a Shared Value: Start by briefly and respectfully introducing yourself and appealing to a value you share with the recipient (e.g., fairness, innovation, public safety, economic prosperity).
2. Clearly State the Issue: Describe the specific application of AI you are concerned about and why you believe it holds both promise and peril.
3. Explain the “Coded Bias” Risk: Using your knowledge of how human biases can be encoded in AI, explain the specific psychological risk. For example, how could Stereotyping lead to discriminatory hiring, or how could the Availability Heuristic be manipulated by recommendation engines?
4. Propose a Principle-Based Solution: Instead of a highly technical fix, propose a broader principle or policy that could help mitigate the risk. This could be a call for more transparency, independent auditing, or a focus on human oversight.
5. End with a Respectful and Forward-Looking Call to Action: Conclude by summarizing your hope for a more responsible and ethical technological future and making a clear, respectful request.

This task requires you to be informed, articulate, and persuasive, all while maintaining a formal and respectful tone.

Grammar Spotlight: Sophisticated Conjunctions and Transitions

To write a formal, well-reasoned argument, you need to move beyond simple conjunctions like “and,” “but,” and “so.” Using more sophisticated conjunctions and transitional phrases will make your writing more fluid and your logic more explicit.

• To Show Contrast or Concession: These words acknowledge an opposing point before making your own.
  • Instead of “but”: try however, nevertheless, nonetheless, yet, on the other hand, conversely, while, whereas, although.
  • Example: “While AI offers the alluring promise of efficiency and impartiality, nevertheless, we must consider the insidious risk of coded bias.”
• To Show Cause and Effect: These words explicitly link a cause to a result.
  • Instead of “so”: try consequently, as a result, therefore, thus, hence.
  • Example: “The training data was based on a historically biased process; consequently, the algorithm learned to replicate those same biases.”
• To Add Information or Emphasize a Point:
  • Instead of “and”: try moreover, furthermore, in addition, indeed, in fact.
  • Example: “The lack of transparency is a significant concern. Moreover, the absence of independent auditing compounds the potential for harm.”
• To Sequence Ideas Logically:
  • Use phrases like First, second, third; The primary concern is…; A further issue is…; In conclusion…

Using a variety of these transitions will elevate your writing from a simple series of statements to a complex, interwoven argument.

Writing Technique: The “Promise, Peril, Principle” Structure

This structure allows you to present a balanced, thoughtful, and forward-looking argument.

1. Promise (Acknowledge the Positive): Start by demonstrating that you are not a Luddite. Acknowledge the potential benefits and the good intentions behind the technology. This builds rapport and shows you are a reasonable person.
   • Example: “I am writing to you as a citizen who is both excited by the potential of artificial intelligence and keenly aware of our shared responsibility to deploy it ethically. The use of algorithmic systems to streamline the hiring process, for example, holds the promise of unprecedented efficiency and, ideally, a more merit-based selection of candidates.”
2. Peril (Explain the Risk): This is the core of your argument. Clearly explain the specific danger of coded bias. Use the concepts and vocabulary from the article to add weight to your explanation.
   • Example: “However, the promise of impartiality can create a dangerous veneer over a more complex reality. As we have seen in cases like Amazon’s experimental recruiting tool, an AI trained on historical data is not learning objective merit; rather, it is learning a digital manifestation of our own past biases. Therefore, a system designed to be impartial can, in fact, become a powerful and insidious vector for perpetuating historical inequities, such as gender or racial stereotyping.”
3. Principle (Propose the Solution): Shift from problem to solution. Propose a high-level principle that should guide development and deployment. This shows you are solution-oriented.
   • Example: “To mitigate this risk, I do not propose that we abandon these powerful tools. Instead, I urge that we adopt a core principle of ‘algorithmic accountability.’ This would mean, first, a commitment to greater transparency about when and how these systems are used. Furthermore, it would necessitate the establishment of independent, third-party audits to scrutinize these systems for discriminatory outcomes before they are deployed. The goal should be to build systems that are not just intelligent, but also equitable and just.”

This structure allows you to build a nuanced argument that is critical without being alarmist, and hopeful without being naive. It is the hallmark of effective public and professional advocacy.

Multiple Choice Quiz

Let’s Discuss

These questions are designed to get you thinking more deeply about the complex intersection of technology, psychology, and society. Use them as prompts for a thoughtful discussion or for personal reflection.

1. Your Algorithmic Life: Think about the algorithms that shape your daily life (e.g., your Netflix recommendations, your Spotify playlist, your social media feed, your Amazon suggestions). In what ways have you seen them create a feedback loop?
   • Dive Deeper: Have you ever noticed one of these systems trying to push you down a particular “rabbit hole”? How has it affected your tastes, your interests, or even your perception of what’s popular or important? Have you ever consciously tried to “retrain” an algorithm by changing your behavior?
2. The Human in the Loop: The article discusses Automation Bias—our tendency to over-trust automated systems. Have you ever followed a GPS into a strange situation or blindly trusted a computer’s suggestion when your own gut told you something was off?
   • Dive Deeper: As AI becomes more integrated into high-stakes fields like medicine and law, what do you think is the right balance between trusting the machine and trusting human expertise? What kind of “human in the loop” systems do we need to prevent a catastrophic failure of automation bias?
3. The Data That Defines You: Algorithms make judgments about you based on the data they can collect. What important aspects of your identity, skills, or potential are invisible in your data trail?
   • Dive Deeper: If an AI were to build a profile of you based only on your online activity, what would it get right? More importantly, what would it get profoundly wrong? How does this make you feel about systems using this data to make decisions about your future (like for jobs or loans)?
4. Solving for Bias: The article suggests solutions like transparency, auditing, and diversifying tech teams. Which of these do you think is most important or would have the biggest impact?
   • Dive Deeper: Can you think of any other potential solutions? For example, what is the role of public education in creating more critical and “algorithm-literate” citizens? Is it the responsibility of companies, the government, or individuals to ensure AI is developed ethically?
5. The Future of “Fairness”: The article argues that an algorithm is an “opinion, embedded in code.” If we have to program our values into our machines, how do we decide whose values to use?
   • Dive Deeper: Consider a self-driving car that must make a split-second decision in a no-win accident scenario. How should it be programmed to decide who to protect? There’s no single right answer. This thought experiment reveals the deep ethical challenges of codifying morality. What does this tell you about the limits of creating a truly “impartial” or “fair” AI?

Learn with AI

Disclaimer:

Because we believe in the importance of using AI and all other technological advances in our learning journey, we have decided to add a section called Learn with AI to add yet another perspective to our learning and see if we can learn a thing or two from AI. We mainly use Open AI, but sometimes we try other models as well. We asked AI to read what we said so far about this topic and tell us, as an expert, about other things or perspectives we might have missed and this is what we got in response.

We’ve had a crucial discussion about how our historical biases get encoded into AI systems, particularly through the data we feed them. We focused on major societal issues like stereotyping in hiring and the spread of misinformation. But I want to zoom in on a more subtle, yet equally powerful, way that algorithms can warp our perception and behavior. Let’s talk about the intersection of algorithms and The Scarcity Mindset.

The Scarcity Mindset is a psychological phenomenon where people, when they perceive a lack of something—whether it’s time, money, or even social connection—tend to have their cognitive capacity significantly reduced. Scarcity captures the mind. It forces you to focus intently on the thing you lack, to the exclusion of almost everything else. This can be good in the short term—it helps you solve the immediate problem—but it has devastating long-term consequences. It narrows your vision, reduces your ability to plan for the future, and diminishes your executive control.

Now, how do algorithms exploit or even induce this mindset?

Think about the design of modern e-commerce and travel websites. You’re looking at a pair of shoes or a hotel room. What do you see? “Only 2 left in stock!” “5 other people are looking at this right now.” “Sale ends in 2:00:00.” These are not neutral pieces of information; they are deliberately designed psychological triggers intended to induce a sense of scarcity.

These triggers hack your brain. They create an artificial sense of urgency and competition. This kicks you out of a “cold,” deliberative state of mind and into a “hot,” scarcity-driven one. In this state, your primary goal is no longer “find the best value” or “make a wise decision.” Your goal is “don’t miss out.” This is a classic sales tactic, but when amplified and personalized by algorithms at a global scale, it creates a constant, low-grade hum of manufactured urgency in our commercial lives.

We see a similar effect on social media platforms with their focus on “streaks” (like on Snapchat) or the fear of missing out (FOMO) on a trending conversation. These platforms are designed to create a sense of time scarcity and social scarcity. The fear of breaking a streak or missing a key moment drives compulsive, repeated engagement. It captures your attentional resources, pulling them away from more important, long-term goals.

The danger is that these algorithmically induced scarcity triggers can have spillover effects. The cognitive load from constantly fighting off FOMO and “limited time offers” can deplete our mental resources, leaving us with less willpower and executive function to handle the truly important decisions in our lives.

So, a crucial element of algorithmic literacy that we didn’t cover is learning to recognize and resist manufactured scarcity. It’s about training yourself to see a “Only 3 left!” banner not as a helpful piece of information, but as a deliberate attempt to activate your scarcity mindset. The antidote is to consciously slow down, to step away from the decision for a few hours, and to ask yourself: “Is this urgent because it’s truly scarce, or because a machine has been designed to make me feel that way?” This act of pausing and questioning the frame is a powerful way to reclaim your cognitive bandwidth from the attention merchants of the digital age.

Let’s Play & Learn

Interactive Vocabulary Building

Crossword Puzzle

Word Search