Redistributed

Paper: Joy Buolamwini & Timnit Gebru. "Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification." Proceedings of Machine Learning Research, 2018. Link: MLR

Why This Paper Matters

In 2018, Joy Buolamwini and Timnit Gebru did something that should have been routine: they tested whether commercial facial recognition systems worked equally well for everyone. The answer was no — not even close.

The paper is foundational AI fairness research, but its significance extends beyond methodology. It demonstrated a structural pattern: AI systems developed in concentrated corporate environments, tested on convenient datasets, and deployed globally distribute their benefits and harms unevenly. The benefits flow to developers and purchasers. The harms accumulate in populations the developers never bothered to test.

The Study

The Problem

Commercial facial recognition systems were being deployed everywhere — airports, police departments, retail stores, social media platforms — without systematic testing for accuracy across demographic groups. The companies selling these systems published impressive headline accuracy numbers. But accuracy for whom?

Buolamwini, then a graduate student at MIT, noticed something troubling: commercial facial recognition systems couldn't reliably detect her face. She had to put on a white mask for the systems to work. This wasn't a bug she encountered. It was a design failure invisible to those who built it.

The Methodology

Existing facial recognition benchmarks had a compositional problem. Popular datasets like LFW (Labeled Faces in the Wild) were disproportionately male and light-skinned, making them useless for measuring disparate performance. You can't find disparities you don't measure.

Buolamwini and Gebru created a new benchmark: Pilot Parliaments Benchmark (PPB). They compiled 1,270 faces of parliamentarians from three African countries (Rwanda, Senegal, South Africa) and three European countries (Iceland, Finland, Sweden). The resulting dataset was balanced:

44.6% female, 55.4% male
46.4% darker-skinned, 53.6% lighter-skinned

They then tested three commercial gender classification systems: Microsoft's Face API, IBM's Watson Visual Recognition, and Face++ (Megvii).

The Findings

The results were stark.

IBM's system:

0.8% error rate on lighter-skinned males
34.7% error rate on darker-skinned females

Microsoft's system:

0.0% error rate on lighter-skinned males
20.8% error rate on darker-skinned females

Face++:

0.7% error rate on lighter-skinned males
34.5% error rate on darker-skinned females

The pattern held across all three vendors: best performance on lighter males, worst performance on darker females. The gap wasn't marginal. For some systems, error rates differed by more than 40 percentage points.

The intersectional analysis was crucial. Looking at gender alone or skin tone alone would have missed the worst disparities. Darker-skinned females faced compounded disadvantage — neither "darker-skinned" nor "female" alone captured their experience.

The Mechanism

How did commercial AI systems end up 40 times worse at identifying some faces than others?

Training data composition. The datasets used to build these systems didn't represent global populations. If your training data is predominantly lighter-skinned males, your system will perform best on lighter-skinned males. This is not bias creeping in; it's bias baked in. As Why Removing Bias From Data Doesn't Remove Bias explores, the problem runs deeper than representation alone.

Benchmark selection. Developers optimized for performance on standard benchmarks, which themselves had demographic skews. If your test looks like your training data, your scores stay high even as your real-world failures accumulate.

Deployment without testing. Companies marketed these systems for global deployment without conducting the demographic testing that would reveal failures. The capability existed to measure disparities. The business incentive didn't.

Who does the testing. The researchers who uncovered these failures were not employed by Microsoft, IBM, or Face++. Buolamwini — a Black woman who personally experienced the systems' failures — did the work that the companies' own engineers could have done but didn't. The labor of proving that AI doesn't work falls on those harmed by it.

The Aftermath

Corporate Response

The paper's impact was immediate. Within months of publication:

IBM substantially reduced error rates and publicly acknowledged the findings. By 2020, IBM announced it would exit the facial recognition business entirely, citing concerns about mass surveillance and racial profiling.

Microsoft improved performance and later called for regulation. President Brad Smith testified before Congress about facial recognition risks, citing accuracy disparities as a key concern.

Face++ initially contested the methodology, then improved performance in subsequent testing.

These responses illustrate a pattern: companies didn't know (or didn't publicize) their systems' failures until external researchers did the work of documenting them. Improvement was possible; it just wasn't prioritized until reputational stakes forced the issue.

Regulatory Impact

Gender Shades catalyzed policy action:

Municipal bans. San Francisco, Oakland, Boston, and other cities banned government use of facial recognition, citing accuracy disparities and civil liberties concerns directly informed by this research.

State legislation. California, Washington, and other states enacted restrictions on facial recognition use, particularly in policing.

EU AI Act. The EU's landmark AI regulation classifies biometric identification as "high-risk," requiring bias testing and documentation. The Act's treatment of facial recognition bears the fingerprints of Gender Shades' findings.

NIST testing. The National Institute of Standards and Technology began publishing demographic breakdowns of facial recognition accuracy, institutionalizing the measurement approach Buolamwini and Gebru pioneered.

The Algorithmic Justice League

Buolamwini founded the Algorithmic Justice League (AJL) to continue this work. AJL has since investigated other systems, advocated for legislation, and created resources for affected communities. The project transformed from a research paper into a sustained institutional challenge to algorithmic harms.

The Redistribution Question

Gender Shades illustrates how AI systems redistribute costs and benefits.

Who paid for the errors? Not Microsoft, IBM, or Face++. Their systems were deployed, their products were sold, their valuations increased. The costs fell on people misidentified — denied services, falsely flagged, subjected to additional scrutiny. In subsequent years, facial recognition failures led to wrongful arrests, all involving Black individuals.

Who did the work? Buolamwini and Gebru built the benchmark, ran the tests, wrote the paper, and faced backlash. Companies with billions in resources didn't test their own products. A graduate student did.

Who benefits from the fix? When systems improve, companies tout their new accuracy numbers. The harms of the previous versions go uncompensated. Those wrongly affected don't get made whole; they just stop being harmed going forward.

This is the pattern Virginia Eubanks documents in Automating Inequality: automated systems concentrate benefits among those who deploy them while distributing costs to populations with the least power to object (Eubanks, 2018). The "digital poorhouse" doesn't malfunction — it functions exactly as designed, for those who designed it.

The Limits of "More Data"

A common response to bias findings is "we just need better data." Gender Shades complicates this.

More representative training data would help. But the problem isn't only data — it's the entire apparatus of development, testing, and deployment. Companies had access to diverse data. They chose not to prioritize it. They had the capability to test for disparities. They chose not to conduct those tests. They had the option to delay deployment until failures were addressed. They chose to ship anyway.

"More data" treats bias as a technical problem solvable by engineers. But the decision not to collect representative data, not to test on diverse populations, not to delay profitable deployment — these are institutional choices, not data gaps. They reflect what companies value.

As Ruha Benjamin argues, technology doesn't just reflect existing inequality — it encodes and amplifies it, creating new mechanisms of discrimination that appear neutral (Benjamin, 2019). Better data within the same institutional structure might reduce disparities without eliminating them.

What Changed — and What Didn't

Gender Shades succeeded in ways most papers don't. It changed corporate behavior, informed policy, and created lasting institutions. The specific systems tested in 2018 have improved substantially.

But the underlying dynamic persists:

AI systems are still developed without adequate demographic testing
The labor of documenting failures still falls on affected communities and independent researchers
Deployment still precedes comprehensive evaluation
Harms still go uncompensated
Commercial incentives still favor speed over equity

Gender Shades didn't solve the problem. It made the problem visible. That visibility created pressure for specific improvements while leaving the structures that produced the failures largely intact.

But visibility is not nothing. The paper demonstrated a model for accountability that has since been replicated across domains: independent audit, public disclosure, institutional response. The Datasheets for Datasets framework, model cards, and the documentation requirements of the EU AI Act all trace lineage to the basic insight Gender Shades established — that external scrutiny produces better systems than internal quality assurance alone. The path from audit to accountability to improvement is now well-documented, even if it remains politically contested.

How to Use This Paper

When arguing that audit matters: Gender Shades demonstrates that internal company testing is insufficient. External, independent evaluation caught failures that vendors' own processes missed — or ignored.

When challenging "neutral" AI: Facial recognition systems weren't designed to be racist. They were designed to perform well on available benchmarks. The racism emerged from mundane institutional choices about what data to collect, what tests to run, what metrics to optimize.

When tracing accountability: The paper maps how harms emerge from distributed decisions — training data choices, benchmark selection, deployment without testing. No single person decided to make systems that fail on Black women. But the cumulative effect of many decisions was exactly that.

When building the case for regulation: The EU AI Act, NIST testing protocols, and municipal bans draw directly on this work. Gender Shades is evidence that voluntary corporate responsibility is insufficient.

Source

Buolamwini, Joy & Gebru, Timnit. "Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification." Proceedings of Machine Learning Research, Volume 81, 2018, Pages 77-91. MLR

General

Paper Review: Gender Shades