Redistributed

Model cards are standardized documentation for AI models. They're supposed to be the nutrition label for machine learning. In practice, they range from genuinely useful to artfully incomplete.

The Original Vision

Mitchell et al. proposed model cards in 2019 as a transparency mechanism (Mitchell et al., 2019). The idea: every model should ship with documentation covering:

What the model is for
How it was trained
How it performs across different groups
Known limitations
Recommended uses

The analogy was to food labels or datasheets for electronics — a concept Gebru et al. later extended to datasets themselves (Datasheets for Datasets). You wouldn't deploy a component without specs; why deploy a model without documentation? For a concrete example of what model documentation looks like from a major lab, see Claude's Model Card.

What You'll Find

A typical model card includes:

Model description — What type of model (language, vision, classification), what architecture, what size. This is usually accurate.

Intended use — What the model was designed for. Often vague ("general-purpose language understanding") or aspirational ("assisting users with various tasks").

Training data — Where the training data came from. Often incomplete ("a mixture of publicly available sources") or hedged ("curated internet text").

Evaluation — Benchmark scores on standard datasets. Numbers that mean something to researchers but little to most users.

Limitations — Known failure modes. Sometimes honest, sometimes perfunctory ("the model may occasionally generate incorrect information").

Ethical considerations — A section required by convention, varying wildly in substance.

What You Won't Find

Specific training data contents. Which websites? Which books? Whose work, used without consent? The data sources that most affect model behavior are usually the least documented.

Real-world performance data. Benchmarks test specific capabilities under controlled conditions. How the model performs on your actual task, with your actual users, isn't measured.

Failure rates by population. "The model performs worse on some groups" is common. "The model misidentifies Black faces at 10x the rate of white faces" is rare, even when true. The Gender Shades study demonstrated exactly this kind of disparity — and it took external researchers to surface it.

Business constraints on design. Why was this capability included or excluded? What shortcuts were taken for cost or speed? The engineering tradeoffs that shape behavior are internal.

Incident history. Has this model (or similar versions) caused harm in deployment? Model cards are typically written at release, not updated with post-deployment failures.

How To Read Them

Start with intended use. If your use case isn't mentioned, you're in uncharted territory. The model wasn't evaluated for what you're doing.

Check the date. Model cards are often created at launch and never updated. A 2022 card for a 2022 model may reflect 2022 capabilities, even if the model has been updated.

Look for specificity. "May generate biased content" is less useful than "generates gender stereotypes in professional contexts at measurable rates." Vagueness often indicates that no one measured.

Notice what's quantified. Numbers suggest someone actually tested something. Qualitative statements ("the model strives to be helpful") are aspirational, not measured.

Read the limitations section last. It's where the real information lives, but it's often written to minimize liability rather than inform users.

The Asymmetry

Model cards are voluntary. There's no standard for what must be included, no penalty for omissions, no third-party verification. They're written by the people who built the model, reviewed by the people who profit from its deployment.

This is like asking pharmaceutical companies to write their own safety warnings without FDA oversight. Some would be thorough and honest. Others would optimize for sales. The EU AI Act moves toward mandatory documentation for high-risk AI systems, and organizations like the Algorithmic Justice League are building infrastructure for external accountability — but these efforts are nascent, and the gap between self-reported model cards and independent verification remains vast.

Better Than Nothing

Despite limitations, model cards are one of the few sources of official documentation for AI systems. A bad model card still reveals something about what the developers thought was important enough to mention — and what they didn't.

The existence of model cards creates an expectation of disclosure. When a model ships without one, that's a signal too.

Sources

Mitchell, Margaret et al. "Model Cards for Model Reporting." FAT, 2019. arXiv
Crisan, Alexandra et al. "Interactive Model Cards: A Human-Centered Approach to Model Documentation." FAccT, 2022. ACM
Raji, Inioluwa Deborah et al. "The Fallacy of AI Functionality." FAccT, 2022. ACM
Gebru, Timnit et al. "Datasheets for Datasets." Communications of the ACM, 2021. arXiv

General

What Model Cards Tell You (And What They Don't)