Dataset
Common Crawl
Open repository of web-crawl data, the largest training source for nearly every LLM.
Definition
Common Crawl publishes petabytes of monthly web snapshots dating back to 2008. Every frontier LLM filters and deduplicates Common Crawl as its primary text source; derived datasets (C4, RefinedWeb, RedPajama) provide cleaner subsets.
Common use cases
- LLM pre-training
- Web research
- Linguistic studies