Redistributed

AI labs have a resource problem. The models keep getting bigger, but the planet's supply of high-quality human-generated text is finite. The entire indexable internet has already been scraped — books, articles, forums, code repositories, academic papers, social media posts, government filings. The well is running dry. So the industry has landed on an elegant-sounding solution: have AI generate its own training data. The trouble is, this solution contains its own undoing.

The Data Wall

Training a large language model requires enormous volumes of text. GPT-3 was trained on roughly 300 billion tokens. GPT-4 consumed substantially more. Each generation demands a larger corpus than the last, and the gains from scaling depend on data quality as much as quantity.

By the early 2020s, labs were already scraping the bottom of the barrel. Epoch AI estimated in 2022 that high-quality text data would be exhausted between 2026 and 2032, depending on growth rates (Villalobos et al., 2022). That timeline has only compressed as models have proliferated. Common Crawl, the backbone of most training sets, covers a vast swath of the web — but "vast" is not the same as "good." Much of it is spam, SEO filler, duplicate content, and machine-generated boilerplate.

The industry's response has been twofold: negotiate licensing deals with publishers and platforms, and manufacture training data synthetically. The first approach is expensive and legally fraught. The second is cheaper but carries risks that are only now becoming clear.

How Synthetic Data Works

The basic idea is straightforward. You take an existing model — a large, capable one — and use it to generate training examples for a new model, or for fine-tuning an existing one. The generated data is filtered for quality, sometimes validated against ground truth, and fed into the training pipeline.

This comes in several flavours. In data augmentation, a model paraphrases or expands existing human-written examples to increase dataset size. In distillation, a large model generates outputs that a smaller model learns to mimic. In self-play, models generate problems and solutions for domains like mathematics or coding, where correctness can be verified programmatically.

Some of these applications work genuinely well — and this is worth emphasising, because synthetic data is not inherently problematic. It has proven effective for mathematical reasoning, where solutions can be checked against known answers. Code generation benefits similarly — you can run the code and see if it works. Google's Gemini and Meta's Llama have both used synthetic data in their training pipelines, particularly for instruction-following and reasoning tasks. Reasoning models like DeepSeek-R1 used synthetic self-play extensively in their training, generating mathematical problems and solutions that demonstrably improved performance.

The problems emerge when you move beyond domains with clear verification criteria.

The Ouroboros Problem

In Greek mythology, the ouroboros is a serpent eating its own tail — an image of self-consuming circularity. It is an apt metaphor for what happens when AI models train on AI-generated data without adequate safeguards.

In 2023, a team led by Ilia Shumailov at the University of Oxford published a paper titled "The Curse of Recursion," which demonstrated the phenomenon they called model collapse (Shumailov et al., 2024). Their findings were stark: when successive generations of models are trained on data produced by previous generations, the output distribution narrows. Rare events and minority perspectives disappear first. The tails of the distribution — the unusual, the creative, the culturally specific — get trimmed with each generation. What remains is an increasingly bland, homogeneous centre.

Think of it like photocopying a photocopy. Each copy is slightly degraded. After enough iterations, you lose the fine detail entirely. The broad strokes remain, but the texture is gone.

The mechanism is not mysterious. A model trained on human data learns a distribution that approximates, but does not perfectly replicate, the true diversity of human expression. When the next model trains on that approximation, it learns a distribution of a distribution — compounding the approximation error. Minority patterns that appeared rarely in the original data appear even more rarely in the synthetic data, and eventually vanish altogether.

This is not just a theoretical concern. The researchers demonstrated it empirically across multiple model architectures, including large language models and variational autoencoders. The degradation was consistent and cumulative.

Where It Works, Where It Fails

The crucial variable is verifiability. Synthetic data works well when you can check whether the output is correct:

Mathematics: Generate problems and solutions, verify the solutions algebraically.
Code: Generate programs, run them against test cases.
Structured data: Generate tabular data that must conform to known schemas and constraints.
Logical reasoning: Generate arguments that can be validated against formal rules.

It fails — sometimes catastrophically — when correctness is subjective, cultural, or context-dependent:

Creative writing: A model generating literary fiction produces fluent text that converges toward a narrow stylistic mean. The range of human literary expression is vast; the range of synthetic literary expression is not.
Cultural content: Models trained on synthetic data lose the markers of lived experience, regional specificity, and cultural nuance that make human-generated text valuable.
Minority perspectives: As Shumailov's work shows, rare viewpoints are the first casualties of model collapse. If only 2% of training data reflects a particular community's language patterns, synthetic regeneration will reduce that percentage further with each cycle.

This last point carries redistributive weight. The communities whose voices are already underrepresented in training data — speakers of minority languages, people with disabilities, indigenous communities — are precisely those whose perspectives vanish fastest under synthetic data regimes.

The Internet Pollution Problem

There is a compounding factor that makes this worse: the web itself is changing. AI-generated content now constitutes a growing share of online text. Studies have estimated that a significant and rising proportion of new web content is machine-generated (Hanley & Durumeric, 2024). Some of it is clearly labelled; much of it is not. Marketing copy, product descriptions, news summaries, social media posts, academic paper mills — the flood of synthetic text is already altering the composition of the internet.

This creates a feedback loop. Future training datasets, scraped from the web, will inevitably contain AI-generated text. Even labs that do not intentionally use synthetic data will consume it inadvertently. The ouroboros eats its tail whether or not you feed it deliberately.

The "dead internet theory" — once a fringe conspiracy — is becoming a partial reality. Not in the sense that all online interaction is fake, but in the sense that the ratio of human-originated to machine-originated content is shifting in a direction that degrades the value of web-scraped training data. The commons is being polluted by the very systems that depend on it. This has implications not just for AI training but for the entire information ecosystem — a concern raised by Bender and colleagues in "Stochastic Parrots".

That said, the research community is actively developing countermeasures. Data provenance tools, watermarking for AI-generated content, and improved filtering techniques are all areas of active development. The problem is real but not yet irreversible.

The Redistribution Angle

Follow the incentives. If synthetic data works well enough, AI labs no longer need to license human-generated content. They can train on their own outputs, reducing their dependence on publishers, writers, artists, and the entire ecosystem of human creators who currently supply the raw material for model training.

This matters because data licensing has been one of the few leverage points that human creators have in the AI economy. The New York Times sued OpenAI in part because licensing fees represent a claim on the value that AI systems extract from human work. Reddit negotiated a $60 million annual deal with Google for training data access. These arrangements, however imperfect, maintain a link between human labour and AI capability.

Synthetic data threatens to sever that link. If a lab can generate its own training data, the economic case for licensing human content weakens. The people who wrote the text that made the first generation of models possible become dispensable — their contribution already absorbed, their ongoing participation unnecessary.

But here is the counterweight — and it is a significant one: if synthetic data degrades quality, there is a floor on how much human labour can be eliminated. The ouroboros problem means that models fed exclusively on synthetic data will converge toward mediocrity. Human-generated data retains value precisely because it carries the diversity, specificity, and unpredictability that synthetic data bleeds away. This gives creators ongoing leverage in the data economy, even as the terms of that leverage shift. Initiatives like Datasheets for Datasets and growing momentum behind data provenance standards could help make that leverage more concrete.

Who benefits: AI labs that can extend model training without paying licensing fees. Companies that can reduce their dependence on expensive, legally contested human-generated datasets.

Who pays: Human creators whose leverage in data negotiations erodes. Users who receive degraded model outputs. Communities whose perspectives disappear from training distributions. The open web, which becomes less reliable as a knowledge source.

What Readers Should Know

Data provenance matters. When evaluating AI tools, ask whether the provider discloses what data was used for training. Models trained heavily on synthetic data may perform well on benchmarks — which tend to test the kind of structured reasoning where synthetic data excels — while failing on the nuanced, creative, and culturally specific tasks where it falls short.

Watch for homogenisation. If AI outputs feel increasingly similar across different tools, synthetic data convergence may be a contributing factor. The bland, confident, mildly helpful tone that characterises many AI outputs is partly an artifact of training pipelines that reward the statistical centre at the expense of the edges.

The quality floor is real. Despite optimistic claims, no lab has demonstrated that purely synthetic training data can match the quality of human-generated data for general-purpose language tasks. The labs that invest in high-quality human data — through licensing, partnerships, or proprietary collection — will likely produce better models. This is worth knowing when choosing which tools to rely on.

The web is changing underneath you. Content you read online is increasingly likely to be AI-generated, whether or not it is labelled as such. This does not make it inherently bad, but it does mean that the informational ecosystem is shifting in ways that affect everyone, not just AI researchers.

Sources

Shumailov, Ilia et al. "AI Models Collapse When Trained on Recursively Generated Data." Nature 631, 2024. Nature
Villalobos, Pablo et al. "Will We Run Out of Data? Limits of LLM Scaling Based on Human-Generated Data." Epoch AI, 2022. arXiv
Hanley, Hans W. A. and Zakir Durumeric. "Machine-Made Media: Monitoring the Mobilization of Machine-Generated Articles on Misinformation and Mainstream News Websites." 2024. arXiv
Alemohammad, Sina et al. "Self-Consuming Generative Models Go MAD." ICLR, 2024. arXiv
Seddik, Mohamed El Amine et al. "How Bad is Training on Synthetic Data? A Statistical Analysis of Language Model Collapse." 2024. arXiv

General

Synthetic Data and the Ouroboros Problem