A nonprofit that maintains an open repository of web crawl data — petabytes of the internet, available to anyone. The invisible foundation beneath nearly every large language model in existence.
What It Is
Common Crawl has been archiving the web since 2008. Every month, they crawl billions of pages and make the data freely available. No login, no fees, no restrictions.
The archive includes:
- Raw web content — HTML, text, metadata
- Monthly crawls — Regular snapshots of the web
- Petabyte scale — One of the largest public datasets
Nearly every large language model has trained on Common Crawl data. It's the foundation of modern AI.
Why It Matters for Redistribution
Who benefits:
- Researchers without corporate budgets — access to web-scale data
- Smaller AI labs — can train models without crawling infrastructure
- Open source projects — foundation for open models like BLOOM, Falcon, MPT
Who pays:
- Common Crawl is funded by donations and grants
- AWS provides free hosting through their Open Data program
- The web itself provides the content (with consent questions unresolved)
What changes: Without Common Crawl, only organizations with massive crawling infrastructure could train language models. The dataset democratizes access to web-scale training data.
The tension: Common Crawl contains copyrighted content, personal information, and problematic material. It's a mirror of the web — with all that entails. The dataset raises hard questions about consent, copyright, and curation that haven't been resolved.
This is the central paradox of open training data. Common Crawl democratizes access to web-scale data — but the data itself was never consented to. Content creators didn't agree to train AI models. As Who Owns the Training Data explores, the question isn't just about access to data but about the rights of those whose work comprises it. The Datasheets for Datasets framework, championed by Timnit Gebru and others, emerged partly in response to datasets like Common Crawl where documentation and provenance were afterthoughts.
Common Crawl also illustrates how How Technology Gets Distributed: a small nonprofit, funded by donations and AWS credits, quietly became critical infrastructure for a trillion-dollar industry. The power asymmetry is stark — the dataset is free, but the models trained on it are not.
Current Status
Active — Common Crawl continues monthly crawls, with data freely available on AWS S3. The organization operates as a nonprofit with a small team.
Key numbers:
- 3+ billion pages per crawl
- 250+ billion pages total
- Petabytes of data
- Used by GPT, LLaMA, Claude, and most major models
How to Get Involved
- Use it: Access data for research and projects
- Donate: Support continued crawling and hosting
- Research: Study the dataset's composition and biases
- Discuss: Engage with governance and consent questions
Related
- BLOOM
- Who Owns the Training Data
- Datasheets for Datasets
- Open Source vs. Open Weights
- How Technology Gets Distributed
- The Frontier Lab Oligopoly
Sources
- Common Crawl. "About." commoncrawl.org
- Common Crawl. "Statistics." commoncrawl.org/stats
- Dodge, Jesse et al. "Documenting Large Webtext Corpora." EMNLP, 2021. arXiv