Dataset
RefinedWeb
TII's 5T-token web-only dataset that powered Falcon — quality from filtering alone.
Definition
RefinedWeb demonstrated that aggressive filtering and deduplication of Common Crawl alone — without curated sources — can produce a 5T-token dataset competitive with the mix-based Pile and RedPajama. Falcon 40B/180B were trained on it.
Common use cases
- Web-only pre-training
- Filtering research