AI Benchmarks Don't Measure What You Think

Model X scores 90.1% on MMLU. Model Y scores 88.7%. Therefore Model X is smarter, more capable, the better choice. This is how AI labs sell their products, how journalists rank models, and how many users pick their tools.

It's also largely nonsense — though not for the reasons you might think, and understanding why is the first step toward evaluating AI systems on terms that actually matter.

Benchmarks measure something. But what they measure and what you need from an AI system are usually different things — and the gap between them is where bad decisions get made, investment gets misallocated, and entire communities get overlooked.

How Benchmarks Work

AI benchmarks are standardized tests. MMLU asks multiple-choice questions across 57 academic subjects. HumanEval checks whether a model can write short Python functions from docstrings. GSM8K tests grade-school math word problems. HellaSwag presents sentence completions that require common-sense reasoning.

Each benchmark produces a score. Labs compile these scores into leaderboard tables, and the tables become the primary currency of AI marketing. Higher numbers mean better model. Simple.

Too simple.

Where It All Breaks Down

To be fair, benchmarks emerged for good reasons. Researchers needed standardized ways to compare models, and early benchmarks drove genuine progress. The problem is not that benchmarks exist — it's that they've been conscripted into a role they were never designed to fill: as comprehensive quality indicators for commercial products. Understanding where they fall short helps us build better evaluation practices.

1. The Memorization Problem

Training a large language model requires vast quantities of text scraped from the internet. Benchmark questions also exist on the internet. The result is what researchers call benchmark contamination: models encounter test questions during training and memorize the answers rather than demonstrating genuine capability.

This isn't hypothetical. Researchers at the Allen Institute for AI found significant contamination across major benchmarks, with some models having seen substantial portions of test sets during training (Sainz et al., 2023). When they tested models on rephrased versions of the same questions — same knowledge required, different wording — performance often dropped significantly.

A model that memorizes the answer to "What is the capital of France?" hasn't demonstrated geographic knowledge. It's demonstrated pattern recall. These are not the same thing, and benchmarks can't tell the difference.

2. Goodhart's Law, on Schedule

"When a measure becomes a target, it ceases to be a good measure." Charles Goodhart formulated this about monetary policy in 1975. It applies to AI benchmarks with uncomfortable precision.

When a lab's funding round depends on beating the competition on MMLU, engineers optimize for MMLU. They adjust training data mixtures, fine-tune on benchmark-adjacent content, and select model configurations that maximize scores on target evaluations. The benchmark stops measuring general capability and starts measuring how well a lab plays the benchmark game.

This isn't cheating, exactly. It's rational behavior under misaligned incentives — the same dynamic explored in The Scaling Laws Plateau, where industry narratives shape capital allocation. But it means the scores tell you less than you think about what the model can actually do for you.

3. Narrow Skill, Broad Claims

A model that aces HumanEval — generating correct Python functions from descriptions — might struggle to debug a real codebase, understand a legacy system, or work within the constraints of a production environment. HumanEval tests a narrow slice of programming ability: writing short, self-contained functions from clear specifications.

Real programming involves ambiguous requirements, sprawling dependencies, trade-offs between speed and readability, and code that has to work alongside other code written by other people over many years. No benchmark captures this — though newer evaluations like SWE-bench are moving in this direction, which represents genuine progress in how we evaluate coding tools.

The same pattern holds everywhere. A model scoring 90% on MMLU isn't demonstrating mastery of 57 academic fields. It's demonstrating the ability to select the right answer from four options on questions that map well to patterns in its training data. Ask it to apply that knowledge in a novel situation — synthesize across domains, handle edge cases, reason under genuine uncertainty — and the benchmark score becomes a poor predictor of performance.

4. The Treadmill

When models saturate a benchmark — all major contenders scoring above 90% — the benchmark stops discriminating between them. MMLU has largely reached this point. So the field invents harder tests: GPQA for PhD-level science questions, SWE-bench for real-world software engineering, ARC-AGI for abstract reasoning.

Each new benchmark is useful for a while. Then models improve, scores converge, and the cycle repeats. François Chollet, creator of the ARC benchmark, has argued that this treadmill reflects a fundamental problem: static benchmarks can't keep pace with rapidly improving systems, and what we really need are evaluations that test for genuine generalization rather than pattern matching at increasing difficulty levels (Chollet, 2019).

The treadmill creates an arms race that consumes research attention without necessarily producing better understanding of what these systems can and can't do. That said, the treadmill also reflects something real: these systems are genuinely improving, and the difficulty of measuring that improvement is itself informative. The challenge is building evaluations that grow with the technology rather than simply being outpaced by it.

5. Strategic Cherry-Picking

Labs choose which benchmarks to highlight. A model that excels at coding benchmarks but underperforms on reasoning tasks will be marketed with coding scores front and center. The benchmarks where it's mediocre quietly vanish from the announcement blog post.

This is standard marketing, no different from a car manufacturer emphasizing horsepower while downplaying fuel economy. But because benchmarks carry the authority of "objective measurement," users treat cherry-picked results as comprehensive assessments. They're not.

6. The Vibes Gap

Here's the most inconvenient problem: user experience often doesn't correlate with benchmark scores.

The LMSYS Chatbot Arena — a crowdsourced platform where users compare anonymous model outputs side by side — has repeatedly surfaced cases where models scoring lower on traditional benchmarks are preferred by actual humans doing actual tasks. Models that "feel" better to use, that follow instructions more naturally, that produce text with better judgment about what the user actually wants — these qualities don't map neatly onto multiple-choice accuracy.

The Chatbot Arena is an improvement over static benchmarks precisely because it measures something closer to what users care about: preference in real use. But it carries its own biases. Its user base skews toward English-speaking, tech-savvy early adopters. The tasks submitted are overwhelmingly in English and reflect the interests of that demographic. It measures what a particular slice of users prefer, not universal quality.

The Saturated Benchmark Problem

When every frontier model scores between 87% and 91% on MMLU, the benchmark tells you almost nothing about which model to choose. A 2-point difference on a saturated benchmark is well within the noise of prompt sensitivity — meaning the same model can score differently based on how questions are framed. As Zheng et al. (2023) demonstrated with the Chatbot Arena, crowdsourced human preference rankings diverge substantially from benchmark orderings once models reach similar capability levels, suggesting that benchmarks lose their predictive value precisely when users need them most.

Yet saturated benchmarks persist in marketing materials because the numbers are high and high numbers look good.

Who Benchmarks Are Built For — and Who They Leave Out

This is where the redistribution frame sharpens the picture.

Benchmarks aren't neutral instruments. They're designed by specific people, in specific institutions, reflecting specific assumptions about what matters. MMLU's 57 subjects are drawn from Western academic curricula. HumanEval tests Python, the lingua franca of Silicon Valley. The vast majority of major benchmarks are English-only.

This means benchmarks systematically overvalue performance on tasks important to their creators and undervalue everything else. A model that excels at MMLU but performs poorly on Yoruba language tasks, or medical triage in low-resource healthcare settings, or agricultural advice for smallholder farmers, will look "better" on the leaderboard than a model optimized for those neglected use cases.

The result is a feedback loop. Benchmarks reward performance on Western, English-language, technically-oriented tasks. Labs optimize for those benchmarks. Investment flows to models that top those leaderboards. Communities whose needs aren't represented in benchmarks get models that weren't built for them — and the gap between AI's promise and its delivery widens along familiar lines of language, geography, and power. This is the same Digital Divide in AI that shapes who benefits from the technology more broadly.

Projects like Masakhane and Mozilla Common Voice represent emerging efforts to build evaluation infrastructure for underserved languages and communities. These initiatives are still small relative to the problem, but they point toward what more representative benchmarking could look like.

As Kate Crawford argues in Atlas of AI, the technical choices in AI systems — including how we measure them — encode political choices about who benefits and who doesn't (Crawford, 2021). Benchmarks are one of the clearest examples: an apparently neutral technical instrument that quietly shapes the distribution of AI's value.

The Benchmark Industrial Complex

There's a secondary economy around benchmarks worth noting. Labs employ teams to optimize benchmark performance. Research papers proposing new benchmarks accrue citations. Leaderboard platforms attract traffic and sponsorship. Benchmark performance drives hiring decisions, acquisition prices, and multi-billion-dollar valuations.

When this much money and prestige rides on benchmark scores, the incentive to produce genuinely useful evaluations takes a back seat to the incentive to produce scores that serve strategic interests. The benchmark stops being a tool for understanding AI systems and becomes a tool for selling them. This dynamic mirrors the broader pattern of how industry narratives shape public understanding of AI capabilities.

The Better Frame

If benchmarks don't tell you what you need to know, what does?

Test on your own tasks. The most reliable way to evaluate an AI model is to run it on your actual work. Not a synthetic benchmark, not a curated demo — your messy, specific, context-dependent tasks. A legal team should test on their real contracts. A customer service operation should test on their actual tickets. A developer should test on their actual codebase.

Look for domain-specific evaluations. Medical AI has clinical trials. Code generation has SWE-bench, which tests against real GitHub issues. These task-specific evaluations, while imperfect, are far more informative than general-purpose benchmarks for specific use cases.

Watch the Chatbot Arena, but skeptically. Crowdsourced preference data is more realistic than automated benchmarks, but it reflects the preferences of its participant pool, not necessarily yours.

Ignore marketing leaderboards. When a lab tells you their model is "#1 on benchmark X," ask: which benchmark? How was it measured? What benchmarks did they choose not to mention? What does that score actually predict about your use case?

Demand transparency. Model cards, red-team reports, and independent evaluations tell you more about a model's real capabilities and limitations than any benchmark score. Initiatives like Datasheets for Datasets are building frameworks for this kind of transparency. If a lab won't share these, their benchmark numbers should inspire less confidence, not more.

What This Means for You

Benchmarks aren't useless. They provide rough signals about a model's general capability level. A model scoring 40% on MMLU is meaningfully different from one scoring 85%. But the difference between 88% and 91%? That tells you almost nothing about which model will serve you better.

The misconception — that benchmark scores are reliable measures of AI quality — persists because it simplifies a genuinely difficult question. Evaluating whether an AI system is good enough for a particular purpose, for particular people, in a particular context, is hard work. Benchmarks offer the illusion of having done that work already.

Don't buy the illusion. Do the work.

Related

• What Model Cards Tell You — The other side of model evaluation: what documentation reveals
• How Language Models Actually Work — Understanding the systems behind the scores
• The Scaling Laws Plateau — When "bigger model, better score" stops holding
• The AI Index Report 2025 — Annual data on how the field actually measures progress
• Datasheets for Datasets — Building transparency into the evaluation pipeline
• Digital Divide in AI — Who gets left out when benchmarks define quality
• Frontier Lab Oligopoly — How benchmark competition reinforces market concentration

Sources

• Sainz, Oscar et al. "NLP Evaluation in Trouble: On the Need to Measure LLM Data Contamination for Each Benchmark." Findings of EMNLP, 2023. ACL Anthology
• Chollet, François. "On the Measure of Intelligence." 2019. arXiv
• Zheng, Lianmin et al. "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena." NeurIPS, 2023. arXiv
• Crawford, Kate. Atlas of AI: Power, Politics, and the Planetary Costs of Artificial Intelligence. Yale University Press, 2021. Publisher
• Goodhart, Charles. "Problems of Monetary Management: The U.K. Experience." Papers in Monetary Economics, Reserve Bank of Australia, 1975.