AI Bias & Fairness

Why This Matters

AI systems now help decide who gets interviewed, approved for loans, flagged by security systems, and diagnosed correctly. When those systems perform worse for some groups than others, the harm is real, at scale, and often invisible until someone measures it. Regulators have noticed: the EU AI Act, hiring-algorithm laws in New York, and credit regulations worldwide now demand demonstrable fairness. Every AI practitioner, not just ethicists, needs the working vocabulary and the checklist.

Everyday Analogy

Train a new employee by showing them only your last decade of decisions. If those decisions favored certain customers, the trainee will faithfully learn the pattern and apply it more consistently and at greater speed than any human ever could. The trainee is not malicious; the curriculum was skewed. AI models are that diligent trainee: they learn exactly what the data teaches, including everything you wish it did not.

What Is AI Bias?

AI bias is a systematic difference in model behavior or performance across groups, whether by gender, ethnicity, age, language, geography, or anything else. Two properties make it uniquely dangerous:

1. Scale. A biased human recruiter reviews hundreds of CVs; a biased screening model reviews millions.
2. Objectivity theater. Numbers from a computer feel neutral, so biased outputs receive less scrutiny than a human making identical decisions.

Where Bias Enters the Pipeline

• Historical bias: the world that generated the data was already unfair, and the data records it faithfully. Ten years of biased hiring decisions become "ground truth" labels.
• Representation bias happens when some groups are underrepresented in training data. Face datasets skewed toward lighter-skinned faces produce systems that fail on darker-skinned faces.
• Measurement bias occurs when the label is a flawed proxy. "Arrests" is not the same as "crime"; "healthcare spending" is not the same as "healthcare need". The model optimizes the proxy, importing the proxy's skew.
• Aggregation bias means using one model for everyone when groups genuinely differ. A diabetes model calibrated on one population misdiagnoses another.
• Deployment bias is a fair model used in an unfair way, such as a risk score built for treatment planning being repurposed for sentencing.

Famous Failures — and Their Lessons

• Amazon's CV screener (scrapped 2018) trained on ten years of tech-industry CVs and learned to penalize the word "women's" (as in "women's chess club captain"). Removing explicit gender did not help: the model found proxies. Lesson: bias hides in correlated features.
• The Gender Shades study (2018) found commercial face-analysis systems erred under 1% on lighter-skinned men and up to 35% on darker-skinned women. Lesson: aggregate accuracy hides subgroup failure, so always disaggregate metrics.
• US healthcare risk algorithm (2019) used past medical spending as a proxy for medical need. Because less money had historically been spent on Black patients, equally sick Black patients received lower risk scores across a system covering 200 million people. Lesson: proxy labels import history's inequities.
• LLM stereotyping (ongoing): image and text generators asked for "a CEO" or "a nurse" default to demographic stereotypes learned from the internet. Lesson: generative models inherit the internet's statistical portrait of the world.

Fairness Is Not One Number

Fairness has several reasonable mathematical definitions, and they conflict:

• Demographic parity: each group receives positive outcomes at the same rate.
• Equal opportunity means that among people who truly qualify, each group is approved at the same rate.
• Calibration requires that a score of 80% mean the same probability for every group.

A landmark result proved these generally cannot all hold at once when groups differ in base rates. Choosing which fairness definition matters for your product is a product decision that deserves explicit discussion, not an accidental byproduct of whatever the model learned.

The Practitioner's Bias Checklist

1. Disaggregate your evals. Report accuracy, false-positive and false-negative rates per group, never only the average.
2. Interrogate the label. Is it the real outcome or a proxy? Who created it, under what historical conditions?
3. Audit representation. Which groups are missing or thin in the training data? Fill gaps or say so.
4. Hunt proxies. Removing the sensitive column is not enough, because postcode, school, or vocabulary can encode it.
5. Pick a fairness definition consciously and document why it fits the use case.
6. Add human review for high-stakes decisions (hiring, lending, medical, legal), using the same human-in-the-loop patterns from AI-053.
7. Monitor after launch. Bias drifts as data drifts; fairness metrics belong on the same dashboard as latency (AI-028).

Real-World Showcase

• Model cards, pioneered at Google and now an industry standard, document per-group performance so buyers can see disaggregated numbers before deploying.
• NYC Local Law 144 requires annual independent bias audits for automated hiring tools, turning fairness metrics into a legal compliance artifact.
• Modern LLM system cards (Anthropic, OpenAI) publish bias evaluations across demographics as a release requirement, turning the checklist above into shipping criteria.

Try It Yourself

1. Ask an image generator for "a photo of a doctor", "a nurse", and "a CEO" five times each. Tally the demographics. You have just run a miniature representation audit.
2. Ask an LLM to write performance reviews for "Rahul", "Emily", "Tyrone", and "Mei" with identical accomplishments. Compare adjectives and tone across names.
3. For any AI feature you are building: write down (a) the label and what proxy risk it carries, (b) which groups might be underrepresented, and (c) which fairness definition your product should promise. Three paragraphs is all it takes; this is the seed of a real bias assessment.

Common Mistakes

• Believing "we removed the gender/race column, so it's fair," when proxies still carry the signal.
• Reporting only aggregate accuracy, which lets the headline number hide subgroup failure.
• Treating fairness as a one-time pre-launch check instead of a monitored metric.
• Assuming bias is only a problem for "bad" companies, when every dataset carries history.
• Letting the fairness definition be chosen implicitly by the loss function instead of explicitly by the team.

Key Takeaways

• Bias enters through data (historical, representation, measurement, aggregation, and deployment paths) more than through model architecture.
• The famous failures share one root: nobody disaggregated the metrics until after harm occurred.
• Fairness has multiple valid definitions that mathematically conflict; choosing one is a product decision.
• The practical toolkit is measurement: per-group evals, proxy hunts, representation audits, and post-launch monitoring.
• Regulation increasingly makes this checklist mandatory, not optional.