- Audio Article
- The Ghost in the Data: How AI Learns to Discriminate
- The High-Stakes Consequences of Coded Prejudice
- Peering Inside the “Black Box”
- Charting a Fairer Course: How We Can Mitigate Bias
- MagTalk Discussion
- Focus on Language
- Vocabulary Quiz
- Let’s Discuss
- Who is ultimately more responsible for algorithmic bias: the people who create the historical data (i.e., society at large) or the tech companies and engineers who build the systems?
- The article discusses the concept of a “black box” AI. How much transparency should we have a right to demand when an AI makes a decision about our lives?
- Imagine a “perfectly fair” hiring algorithm exists. It shows no bias based on race, gender, age, etc. What might be the unintended negative consequences of using such a system?
- The article mentions using diverse teams to help mitigate bias. Besides demographic diversity (race, gender), what other types of diversity are crucial to have on a team building AI?
- Should a company be legally liable if its AI is found to be discriminatory, even if the company can prove it did not intend for the system to be biased?
- Learn with AI
- Let’s Play & Learn
Audio Article
We tend to think of computers as paragons of objectivity. They are machines of pure logic, untouched by the messy, irrational prejudices that so often cloud human judgment. We put our data in, and they give us an impartial answer out. We hold up artificial intelligence as a mirror to our world, hoping to see a reflection of pure, unadulterated fact. But what if the mirror is flawed? What if it’s a funhouse mirror, one that takes the world we show it and reflects back a distorted, exaggerated, and deeply unfair version of reality?
This is the subtle but deeply concerning problem of algorithmic bias. As we increasingly delegate consequential decisions to automated systems—from who gets a job interview to who gets a loan, and even who might be a future criminal—we are discovering that these systems are not the neutral arbiters we hoped they would be. Instead, they are often learning, perpetuating, and in some cases, amplifying the very worst of our societal biases. The problem isn’t that the machines have suddenly developed prejudices of their own. The problem is the mirror itself. The algorithms are holding a reflection up to the data we provide, and that data, drawn from our messy, unequal human history, is anything but impartial. Unmasking the flaws in our digital mirrors has become one of the most pressing ethical challenges of the 21st century.
The Ghost in the Data: How AI Learns to Discriminate
To understand algorithmic bias, you first have to understand, in a very basic sense, how modern AI learns. Most of the AI systems in use today are powered by a process called machine learning. You don’t program them with explicit rules, like “If a person has a college degree, then they are a good job candidate.” Instead, you show them millions of examples and let them figure out the patterns for themselves. It’s like teaching a child to recognize a cat by showing them thousands of pictures of cats, not by describing a cat’s features.
Now, imagine you want to build an AI to screen résumés for a top engineering firm. The most logical approach is to feed it the résumés of every engineer the company has hired over the last 20 years and tell it, “Find more people like this.” The AI diligently gets to work, analyzing every data point. It learns the patterns that correlate with success at the company. But what if, for those 20 years, the company predominantly hired male engineers? What if the hiring managers, consciously or unconsciously, favored candidates from certain universities or neighborhoods?
The AI won’t know about sexism or classism. It will only see patterns. It might learn that résumés containing words like “women’s chess club captain” are negatively correlated with being hired. It might learn that applicants named Jared or John are more likely to be successful than applicants named Maria or Aisha. It will learn the company’s historical bias as a success strategy and then apply it with ruthless, mathematical efficiency. It has, in effect, automated the company’s prejudice. This is the core of the problem: the data we feed these systems is a fossil record of our past decisions, complete with all our historical inequalities and ingrained biases. When we use that data as a blueprint for the future, we risk entrenching those old injustices in a new and powerful technological framework.
The High-Stakes Consequences of Coded Prejudice
This isn’t just a theoretical problem. The impact of algorithmic bias is being felt by real people in some of the most critical areas of their lives. The digital gatekeepers are already here, and often, their gates are unfairly closed.
The Automated Gatekeepers: Bias in Hiring and Loans
One of the most famous cautionary tales comes from Amazon. In 2014, the company began building an AI tool to automate its résumé screening process. The team fed it a decade’s worth of résumés submitted to the company. As discussed above, because the tech industry has been historically male-dominated, the model taught itself that male candidates were preferable. It learned to penalize résumés that contained the word “women’s” and even downgraded graduates of two all-women’s colleges. Amazon’s engineers tried to edit the system to make it more neutral, but they couldn’t guarantee it wouldn’t find new, more subtle ways to discriminate. The project was ultimately scrapped.
The world of finance is another minefield. Lenders are increasingly using complex algorithms to decide who gets approved for a mortgage, a car loan, or a credit card. These systems are forbidden by law from using protected attributes like race. But they can use a thousand other data points that act as a proxy for race. A proxy is a substitute variable that is closely correlated with the one you’re not supposed to use. For example, an algorithm might learn that applicants from certain zip codes, with certain spending habits, or who shop at particular stores are higher credit risks. If those zip codes are historically segregated due to decades of discriminatory housing policies, the algorithm has effectively rediscovered redlining—the illegal practice of denying services to residents of certain areas based on their race or ethnicity. It’s discrimination with a digital facelift.
The Code of Justice: Unfairness in the Courtroom
The stakes are perhaps highest in the criminal justice system. Across the United States, judges use AI-driven risk assessment tools to predict the likelihood that a defendant will re-offend. One of the most widely used and controversial tools, COMPAS, was the subject of a now-famous 2016 ProPublica investigation. The journalists found that the algorithm was starkly biased against Black defendants. They were almost twice as likely as white defendants to be incorrectly labeled as having a high risk of re-offending. White defendants, conversely, were more often mislabeled as low risk.
The creators of the software argued that the tool was not biased because it was equally accurate in predicting recidivism for both Black and white defendants. This kicked off a fiendishly complex debate about the very definition of “fairness.” Is an algorithm fair if it’s equally accurate overall but makes different types of errors for different racial groups? A human parole board might be able to weigh the unique circumstances of an individual’s life, but an algorithm sees only the data it was trained on—data that reflects a history of unequal policing and sentencing. The result is a system that can create a devastating feedback loop: the algorithm predicts a certain group is higher risk, leading to harsher sentencing, which in turn generates more data confirming the initial biased prediction.
Peering Inside the “Black Box”
Complicating all of this is a problem that even stumps the AI developers themselves: the “black box” phenomenon. With many of today’s most powerful AI models, particularly deep learning networks, the decision-making process is profoundly opaque. The system ingests vast amounts of data and, through millions of calculated adjustments to its internal parameters, learns to produce an output. But the path it takes from input to output—the “why” behind its decision—can be nearly impossible for a human to retrace or understand.
Imagine a hiring algorithm that rejects a candidate. The company might not be able to provide a specific reason. The AI didn’t follow a simple checklist; it recognized a complex, multi-dimensional pattern across hundreds of data points that it had associated with unsuccessful candidates. It can’t explain its reasoning in human terms. This is a massive problem. How can you appeal a decision if you don’t know why it was made? How can a company fix a biased system if its own creators don’t fully understand how it works? It creates a world of unaccountable, unexplainable authority.
In response to this, a major push is underway in the tech community to develop “Explainable AI” (XAI). The goal of XAI is to create systems that can justify their decisions in a way that is understandable to humans. An XAI system wouldn’t just deny a loan application; it would highlight the primary factors that led to its decision, such as “high debt-to-income ratio” or “short credit history.” This transparency is crucial. It allows for auditing, for debugging, for identifying and correcting bias, and for providing a mechanism for people to challenge automated decisions that affect them. It’s a movement to replace the black box with a glass box.
Charting a Fairer Course: How We Can Mitigate Bias
The challenge of algorithmic bias is formidable, but it is not insurmountable. We are the ones who build these systems, and we have the power to build them better. The path forward requires a multi-faceted approach involving technologists, policymakers, and the public.
The Data Janitors and Algorithmic Auditors
Since the root of the problem is often the data itself, one of the most critical steps is rigorous data “hygiene.” This involves carefully auditing datasets before they are used to train a model. Are certain demographic groups underrepresented? Does the data reflect historical biases? This work, sometimes called “data janitoring,” is unglamorous but absolutely essential. Beyond just cleaning the source data, independent algorithmic auditors can be brought in to test the AI system itself for biased outcomes, much like a financial auditor inspects a company’s books. They can run simulations, test the system with different demographic profiles, and identify where the model is behaving unfairly.
The Power of Diverse Teams
Technology is not created in a vacuum; it is shaped by the values and perspectives of the people who build it. A homogenous team of developers, all from similar backgrounds, is far more likely to have blind spots that can lead to biased systems. They might not think to test how a facial recognition system performs on darker skin tones or how a voice assistant interprets different accents. Conversely, a diverse team—with members from different genders, races, ethnic backgrounds, and socioeconomic statuses—brings a wider range of lived experiences to the table. They are better equipped to anticipate potential problems, question assumptions, and build technology that works for everyone, not just for the group that designed it.
The Digital Referees: Regulation and Oversight
Finally, we cannot rely solely on the goodwill of tech companies to solve this problem. Just as we have regulatory bodies like the Food and Drug Administration (FDA) to ensure our food and medicine are safe, there is a growing call for government-led regulation to ensure our algorithms are fair. This could take the form of mandatory transparency requirements, where companies must be able to explain how their high-stakes AI systems work. It could involve regular, mandated audits for any algorithm used in public services or critical sectors like finance and hiring. The goal is not to stifle innovation, but to create a set of rules for the road that ensures the immense power of AI is harnessed for the public good, not to its detriment.
The algorithms we build are a mirror. For now, that mirror often reflects a distorted and unequal society. But the reflection is not our destiny. By consciously and deliberately working to clean the data, diversify the creators, demand transparency, and establish clear rules, we can begin to shape that mirror. We can build algorithms that reflect not the flawed world as it has been, but the fair and equitable world we hope to create.
MagTalk Discussion
Focus on Language
Vocabulary and Speaking
Let’s dive into some of the language from that article. It’s easy to get lost in the technical jargon, but a lot of the most powerful words we used are actually incredibly versatile and can make your everyday English sound much more sophisticated and precise. Let’s start with a pair of words that are really at the heart of the issue: perpetuate and entrench. In the article, we talked about how flawed data perpetuates and even entrenches societal inequalities. To perpetuate something means to make it continue indefinitely, to keep it going. Think of it like passing a flame from one candle to another. You are perpetuating the fire. In a social context, you might say, “Stereotypes in movies often perpetuate harmful ideas about certain groups of people.” It means the movies are causing these bad ideas to continue. Entrench is similar, but it’s even stronger. To entrench something is to establish it so firmly that it’s very difficult to change. Imagine a soldier digging a trench in the ground. They are making a deep, solid, and defensible position. When a belief or a system becomes entrenched, it’s like it’s been dug into the fabric of society. You could say, “After decades of the same policies, these economic problems have become deeply entrenched and will be hard to solve.” So, AI doesn’t just keep bias going (perpetuate), it can make it deeper and harder to remove (entrench).
Next up is the word subtle. We described algorithmic bias as a subtle but concerning problem. Subtle means not obvious, and therefore, difficult to notice or describe. It’s the opposite of “loud” or “in your face.” A subtle flavor in a dish is one you have to pay close attention to notice. A subtle hint is one that isn’t direct. This word is perfect for algorithmic bias because the discrimination isn’t happening because of a sign that says “No women allowed.” It’s happening deep inside a complex system, through patterns that are hard to see. In daily conversation, you might say, “There was a subtle change in her tone of voice that made me think she was upset.” Or, “I appreciate the subtle humor in his writing; it’s clever without being obvious.”
Let’s talk about consequential. We mentioned that we’re delegating consequential decisions to AI. This is a fantastic word that simply means important or significant. It has the word “consequence” inside it, which is a great clue. A consequential decision is one that will have major consequences or results. Deciding what to eat for lunch is usually not a consequential decision. Deciding whether to move to a new country is highly consequential. You could say, “The invention of the printing press was one of the most consequential events in human history.” It’s a more formal and powerful way of saying “very important.” A word that goes hand-in-hand with this is insidious. While we didn’t use this one in the article, it’s the perfect partner to subtle. Insidious means proceeding in a gradual, subtle way, but with harmful effects. It’s a negative word. A disease can be insidious, starting with minor symptoms but becoming very dangerous over time. Rumors can be insidious, spreading quietly until they’ve ruined someone’s reputation. Algorithmic bias is insidious because it works quietly behind the scenes, and you might not even realize the damage it’s doing until it’s too late. You might say, “The insidious effects of sleep deprivation can harm your health over time.”
Now for a more technical, but increasingly common word: opaque. We described a “black box” AI as being opaque. Literally, opaque means not able to be seen through; not transparent. A frosted glass window is opaque. We use it metaphorically to describe anything that is hard or impossible to understand. Complicated legal documents can be filled with opaque language. A person who doesn’t show their emotions can be described as opaque. When we say an AI’s decision-making process is opaque, we mean we can’t see inside it to understand its logic. It’s a great word to use when you want to say something is confusing or deliberately unclear. “The company’s new privacy policy is completely opaque; I have no idea what they’re doing with my data.”
To fight against this opacity, we need to scrutinize these systems. To scrutinize something is to examine it thoroughly and carefully. It’s not just a quick glance; it’s a deep, critical inspection. A jeweler will scrutinize a diamond for flaws. A proofreader will scrutinize a text for errors. It implies you’re looking for problems or trying to find the truth. You might say, “Before signing the contract, my lawyer will scrutinize every clause.” It’s a strong verb that shows you are being very diligent and careful. Once we scrutinize the problem, we need to mitigate it. To mitigate something means to make it less severe, serious, or painful. You can’t always eliminate a problem entirely, but you can often mitigate its effects. For example, planting trees can help mitigate the effects of air pollution. Taking painkillers can mitigate the pain of a headache. In the article, we talked about steps to mitigate bias. We might not be able to create a perfectly unbiased AI, but we can take actions to make the bias less harmful. It’s a very useful word in professional settings: “Our new safety protocols are designed to mitigate the risk of accidents in the workplace.”
Finally, let’s look at the word proxy. We said that an algorithm might use a person’s zip code as a proxy for their race. A proxy is a figure that can be used to represent the value of something in a calculation. More simply, it’s a substitute or a stand-in. In a company meeting, if your boss can’t attend, they might send you as their proxy to vote on their behalf. You are their stand-in. In data science, a proxy is a variable that isn’t directly relevant on its own, but that is closely correlated with the variable you’re interested in (or not allowed to use). It’s a powerful concept because it shows how systems can discriminate indirectly. In a less technical sense, you could say, “Some people argue that a person’s taste in music can be a proxy for their political beliefs.” You’re suggesting it’s an indirect indicator.
So now we have this great set of words: perpetuate, entrench, subtle, consequential, insidious, opaque, scrutinize, mitigate, and proxy. How do we use them to sound more articulate when we speak? A key skill in persuasive speaking is stating a problem and then proposing a solution. This structure makes your argument clear, logical, and compelling. Let’s practice that. Imagine you’re in a meeting discussing a new company policy. You think the policy has a hidden flaw. You could say: “My concern is that this new policy, while it seems fair on the surface, could have some subtle and insidious effects. I’m worried it might inadvertently perpetuate the communication gap between our departments. The language in section three is particularly opaque, which could lead to confusion. This is a consequential decision, so I believe we need to scrutinize the potential outcomes more carefully before we implement it. Perhaps we can form a small committee to suggest changes that would mitigate these risks.”
Do you see how that flows? You identified a subtle, insidious problem that could perpetuate an existing issue. You pointed to a specific part that was opaque. You stated the stakes by calling it consequential. You proposed a course of action (scrutinize) and an ultimate goal (mitigate). This structure is useful everywhere, from work presentations to arguing with a friend about which movie to watch.
Here’s your speaking challenge. I want you to think of a problem, big or small, in your community, your workplace, or even just in your daily routine. It could be anything from traffic congestion to an inefficient process at work. Your task is to describe this problem and propose a solution in a short, 60-second speech. Try to use at least three of the vocabulary words we’ve discussed today. Use that “Problem/Solution” structure. Start by explaining why the problem is consequential or how its effects are insidious. Then, propose a way to mitigate the problem. Record yourself if you can. This practice will help you organize your thoughts on the fly and use sophisticated vocabulary to make a clear and persuasive point.
Grammar and Writing
Welcome to the writing section. Today, we’re going to tackle a writing challenge that puts you right in the middle of the algorithmic bias dilemma. It’s one thing to understand the issue, but it’s another to articulate a clear, persuasive argument about it. Good writing here isn’t just about avoiding spelling mistakes; it’s about building a logical case and using the right tone to achieve your goal.
The Writing Challenge
Here is your writing prompt:
You have been rejected for a significant opportunity (e.g., a mortgage, a specialized job, or a university admission) and you have strong reason to believe the decision was made by an automated AI system and may have been biased. Write a formal letter (400-600 words) to the organization’s Head of Review or a similar department. In your letter, you must:
- Clearly state the purpose of your letter.
- Explain why you believe the decision was flawed, referencing the possibility of algorithmic bias without being overly accusatory.
- Request a human review of your application and greater transparency regarding their decision-making process.
- Maintain a professional, respectful, and firm tone throughout.
This is a tricky balancing act. You need to be assertive but not aggressive. You’re making a serious claim about bias, but you don’t have definitive proof. Your language must be precise, your tone must be measured, and your structure must be logical.
Let’s break down some tips and grammar structures that will be essential for writing a successful letter.
Tip 1: Master the Art of the Formal, Diplomatic Tone
When you’re writing a letter of complaint or appeal, your tone is everything. If you sound angry and accusatory, the reader may become defensive. If you sound too passive, your request may be dismissed. The sweet spot is a tone of firm, respectful inquiry.
A key grammatical tool for achieving this tone is the passive voice. Now, you’ve probably been told to avoid the passive voice (“The ball was hit by John” instead of “John hit the ball”). In creative or direct writing, that’s often good advice. But in formal, diplomatic, or scientific contexts, the passive voice is your friend. It allows you to emphasize the action rather than the actor, which can make your statements sound less personal and accusatory.
- Active (Accusatory): “Your algorithm rejected my application.”
- Passive (Formal, Objective): “My application was rejected.”
- Active (Accusatory): “You made a decision based on flawed data.”
- Passive (Objective): “It appears a decision was made based on what may be flawed data.”
Notice how the passive construction shifts the focus from “you” to the decision itself. This is less confrontational and invites a more collaborative, problem-solving response.
Grammar Deep Dive: The Passive Voice
The passive voice is formed using a form of the verb to be + the past participle of the main verb.
- Present Simple Passive: “A review is requested.”
- Past Simple Passive: “My application was reviewed.”
- Present Perfect Passive: “A mistake has been made.”
- Modal Passive: “The data should be scrutinized.”
In your letter, use the passive voice strategically in your opening sentences (“I am writing in regard to the decision that was made on my application…”) and when you are referring to the negative outcome.
Tip 2: Speculate Intelligently with Modal Verbs
You don’t know for sure that an algorithm was biased against you. You only suspect it. This is where modal verbs of possibility, probability, and deduction are essential. They allow you to put forward your theory without stating it as an absolute fact.
- Instead of: “The algorithm is biased.”
- Use: “It is possible that the algorithm may be unintentionally biased.”
- Instead of: “The system discriminated against me because of my background.”
- Use: “I am concerned that the system could have inadvertently used proxy variables that correlate with my background.”
- Instead of: “This was an error.”
- Use: “It seems there might have been an error in the evaluation process.”
Grammar Deep Dive: Modal Verbs of Deduction and Possibility
These verbs express your level of certainty about something.
- High Certainty (Deduction): must have / can’t have
- (You probably won’t use this, as it’s too strong, but it’s good to know.) “Given my qualifications, there must have been a misunderstanding.”
- Medium Certainty / High Possibility: could have / may have
- “The automated screening process may have overlooked key qualifications in my résumé.”
- “It’s conceivable that the data used to train the system could have contained historical biases.”
- Low Possibility: might have
- “I wonder if my non-traditional career path might have been misinterpreted by the system.”
Using these modals makes you sound thoughtful and reasonable. You are not making accusations; you are raising valid concerns and opening a door for investigation.
Tip 3: Build a Logical Case with Cause-and-Effect Language
Your letter is an argument. You need to connect your qualifications (the cause) to the illogical outcome of your rejection (the effect) and suggest a possible reason for the disconnect (algorithmic bias). Use transition words and phrases that clearly signal these cause-and-effect relationships.
Here’s how you can structure a paragraph:
- State your strength (Cause): “My application highlighted over ten years of experience in project management, consistently exceeding targets by an average of 15%.”
- State the surprising outcome (Effect): “Consequently, I was surprised and disappointed to learn that my application was not selected to move forward.”
- Propose the potential intervening factor: “Given that my qualifications appear to align closely with the role’s requirements, I am led to believe my application may not have been fully evaluated by a human reviewer. This has raised concerns for me about the possibility that an automated system, due to its reliance on historical data, might have unfairly penalized my profile.”
Grammar Deep Dive: Cause-and-Effect Conjunctions and Transitions
- To show a result: consequently, as a result, therefore, thus, for this reason
- “My qualifications are exceptional; therefore, the rejection seems anomalous.”
- To explain a reason: due to, because of, owing to, as a result of
- “Due to the opaque nature of automated decisions, I am requesting more information.”
- To introduce a basis for your thinking: given that, seeing as, in light of
- “In light of my extensive experience, the outcome is difficult to understand.”
By combining a formal passive voice, intelligent speculation with modal verbs, and a logical structure built with cause-and-effect language, you can write a letter that is powerful, persuasive, and professional. You are showing that you are a serious, thoughtful candidate who deserves a second look.
Vocabulary Quiz
Let’s Discuss
Who is ultimately more responsible for algorithmic bias: the people who create the historical data (i.e., society at large) or the tech companies and engineers who build the systems?
Think about where the blame lies. Is it fair to blame an engineer for building a system that accurately reflects a biased world? Or do they have an ethical obligation to actively correct for those biases, even if it means the system no longer perfectly reflects the raw data? Discuss the shared responsibility between society, which produces the biased “fossil record,” and the creators of technology, who have the power to either replicate or reshape it.
The article discusses the concept of a “black box” AI. How much transparency should we have a right to demand when an AI makes a decision about our lives?
Consider the trade-offs. Full transparency might reveal a company’s proprietary secrets or be too complex for a layperson to understand. On the other hand, a complete lack of transparency can feel unjust and arbitrary. Where is the line? Should there be a “right to an explanation” for any consequential automated decision? How would that work in practice for something like a loan or job application?
Imagine a “perfectly fair” hiring algorithm exists. It shows no bias based on race, gender, age, etc. What might be the unintended negative consequences of using such a system?
Think beyond the obvious benefits. Could a perfectly “fair” system that optimizes only for stated qualifications lead to a workforce where everyone has the same type of background and education, killing diversity of thought? Could it devalue hard-to-measure qualities like creativity, resilience, or potential for growth? Discuss whether our human “biases” are always negative, or if they sometimes allow for intuition and the ability to see potential that data alone can’t capture.
The article mentions using diverse teams to help mitigate bias. Besides demographic diversity (race, gender), what other types of diversity are crucial to have on a team building AI?
Brainstorm different dimensions of diversity. What about diversity of academic background (e.g., having philosophers and sociologists on the team, not just computer scientists)? What about diversity of socioeconomic background, political thought, or geographic origin? Discuss how each of these different perspectives could help a team spot a potential bias that others might miss.
Should a company be legally liable if its AI is found to be discriminatory, even if the company can prove it did not intend for the system to be biased?
This is a core legal and ethical question. Does intent matter, or only the outcome? Compare it to other areas of law. For example, a company is liable if its factory pollutes a river, even if it didn’t intend to. Should the same standard of strict liability apply to “algorithmic pollution”? Discuss the pros and cons of holding companies responsible for the actions of their opaque, complex systems.
Learn with AI
Disclaimer:
Because we believe in the importance of using AI and all other technological advances in our learning journey, we have decided to add a section called Learn with AI to add yet another perspective to our learning and see if we can learn a thing or two from AI. We mainly use Open AI, but sometimes we try other models as well. We asked AI to read what we said so far about this topic and tell us, as an expert, about other things or perspectives we might have missed and this is what we got in response.
It’s great to have a moment to add a couple of extra layers to this conversation. The article provided a fantastic overview of the problem and some of the mainstream solutions being discussed. But I want to touch on two aspects that often get lost in the shuffle: the nature of “fairness” itself, and the risk of what I’ll call “fairness-washing.”
First, let’s talk about fairness. The article rightly points out the debate around the COMPAS algorithm, where one side said it was biased and the other said it was “equally accurate.” This highlights a fundamental, almost philosophical, problem in this field: there is no single, universally accepted mathematical definition of fairness. In fact, computer scientists have proven that some of the most common and intuitive definitions of fairness are mutually exclusive. You literally cannot satisfy all of them at the same time.
For example, one definition of fairness is “demographic parity,” which means the algorithm’s approvals should match the demographics of the applicant pool. If 50% of your loan applicants are women, 50% of your loan approvals should go to women. That sounds fair. But another definition is “calibration,” which means that for any given risk score the algorithm produces, the actual rate of default should be the same regardless of group. If the algorithm gives you a score of 700, your probability of paying back the loan should be the same whether you are a man or a woman. This also sounds fair. The problem is, unless the underlying rates of default are already identical between men and women in the real world, you cannot build one algorithm that satisfies both of these fairness criteria simultaneously.
What does this mean in practice? It means there is no purely technical fix for bias. Deciding which definition of fairness to optimize for is not a mathematical choice; it’s an ethical and political choice. It requires us to have a conversation as a society about our values. Do we want to prioritize equality of outcome, or do we want to prioritize equal predictive accuracy? We can’t just hand this problem to the engineers and tell them to “make it fair.” We have to give them a specific definition of what we mean by fair, and that is a conversation that needs to include everyone.
The second point is about “fairness-washing.” This is a term similar to “greenwashing,” where a company spends more time and money marketing itself as environmentally friendly than on actually minimizing its environmental impact. In the AI world, fairness-washing is when a company makes a big public show about its commitment to ethical AI and fighting bias, but its internal practices don’t match its rhetoric.
This can look like a company publishing a vague set of “AI Ethics Principles” that sound nice but have no enforcement mechanism. It can look like hiring a few “AI Ethicists” but giving them no real power to stop the launch of a problematic product. One of the most subtle forms of fairness-washing is over-emphasizing the “unconscious bias training” for employees while under-investing in the hard, expensive work of auditing datasets and redesigning systems. It’s much easier to host a workshop than it is to fundamentally change a profitable but biased product. As consumers and citizens, we need to look past the press releases and ask the hard questions: What is your audit process? Who is on your ethics board, and what power do they have? Can you explain the decisions of the algorithms you use on us? The demand for genuine action, not just ethical branding, is one of the most powerful tools we have to push for real change.
0 Comments