Redistributed

General

Why Removing Bias From Data Doesn't Remove Bias

Published March 7, 2026

pitfallbiasfairness

The most common response to AI bias problems is "we need to fix the training data." This sounds reasonable — and data quality genuinely matters. But treating it as the complete solution is insufficient, sometimes counterproductive, and often a distraction from harder, more productive questions. Understanding why helps us build fairer systems, not just cleaner datasets.

The Intuition That Misleads

The logic seems obvious: AI learns from data. If the data contains bias, the AI will learn bias. Therefore, remove bias from the data, and you'll get unbiased AI.

This framing treats bias as a contaminant — something that can be filtered out, leaving behind pure, neutral data. But bias isn't a substance. It's a relationship between a system and its context of use.

Where the Model Breaks Down

Consider a hiring algorithm trained on historical hiring decisions. If past hiring was discriminatory, the data reflects that discrimination. The usual response: remove protected characteristics (race, gender, age) from the training data.

This fails for several reasons:

Proxy variables exist. Zip codes correlate with race. Names correlate with gender. University names correlate with socioeconomic background. The model can reconstruct protected categories from unprotected features. Removing explicit labels doesn't remove the information (Barocas & Selbst, 2016). This is the same pattern Joy Buolamwini documented in Gender Shades — the data reflects structural patterns that persist even after surface-level corrections.

Historical patterns persist. Even with balanced data, the model learns what "success" looked like historically. If success was defined by the biased judgment of past decision-makers, the model inherits that definition. You can have perfectly representative data that teaches the model to replicate injustice.

Sampling bias compounds. Who's in the training data in the first place? If certain groups were excluded from opportunities, they're underrepresented in "success" examples. The model learns that success looks like the people who historically had access.

The Deeper Problem

"Unbiased data" implies there's a neutral, objective ground truth that data can capture. For many consequential decisions — who should get a loan, who should get parole, who should get a job — there is no ground truth. There are only human judgments, embedded in historical and social contexts.

When we train a model to predict "creditworthiness," we're training it to predict past lending decisions. Those decisions were made by humans with their own biases, in institutional contexts with their own incentives, using criteria that reflected the values of their time and place.

The data isn't biased relative to some ideal — the data is the record of biased human decisions. As Ruha Benjamin argues, technology doesn't just reflect existing inequality — it actively encodes and amplifies it, creating what she calls "the New Jim Code" (Benjamin, 2019). Virginia Eubanks's Automating Inequality documents this dynamic in detail across welfare, housing, and criminal justice systems.

What Actually Helps

If data cleaning isn't sufficient, what is?

Define fairness explicitly. "Unbiased" is meaningless without specifying biased-relative-to-what. Different fairness criteria (equal opportunity, demographic parity, individual fairness) are mathematically incompatible — a point Corbett-Davies & Goel (2018) demonstrate rigorously. You have to choose, and choosing is a value judgment. Moreover, as Selbst et al. (2019) argue, fairness cannot be abstracted away from the social context in which systems operate. Making these choices transparent — through model cards and datasheets — is a necessary first step.

Measure outcomes, not inputs. Rather than trying to create "clean" data, measure how the system actually affects different groups in deployment. Does it produce disparate impacts? By how much? Is that acceptable? These are policy questions, not engineering questions.

Question the task itself. Sometimes the right response to "how do we build an unbiased recidivism predictor?" is "should we be predicting recidivism at all?" The assumption that a decision should be automated often goes unexamined.

Involve affected communities. The people impacted by a system often see failure modes that designers miss. Participatory design isn't just ethical window dressing — it produces better systems. Organisations like the Algorithmic Justice League and the DAIR Institute are developing frameworks for this kind of community-centered accountability.

The Infrastructure Implication

The "just fix the data" framing treats fairness as a technical problem that engineers can solve. This conveniently keeps decision-making inside tech companies and away from democratic deliberation.

But decisions about who bears the costs of AI errors, which tradeoffs are acceptable, and what "fairness" even means in context are not technical questions. They're political questions. The infrastructure of AI governance — who decides, using what process, with what accountability — matters more than any particular debiasing technique.

Data is not the problem. The problem is mistaking a political question for a technical one. The good news is that better frameworks exist — ones that combine technical rigour with democratic input. The challenge is building institutions that use them. Democratic AI Governance and AI and Democracy explore what those institutions might look like.

Sources

  • Barocas, Solon & Selbst, Andrew. "Big Data's Disparate Impact." California Law Review, 2016. SSRN
  • Corbett-Davies, Sam & Goel, Sharad. "The Measure and Mismeasure of Fairness." Journal of Machine Learning Research, 2018. arXiv
  • Benjamin, Ruha. Race After Technology. Polity, 2019. Publisher
  • Selbst, Andrew et al. "Fairness and Abstraction in Sociotechnical Systems." FAT 2019. ACM
← Back to all articles
On this page