Dataset
RefinedWeb
TII's 5T-token web-only dataset that powered Falcon â quality from filtering alone.
Definition
RefinedWeb demonstrated that aggressive filtering and deduplication of Common Crawl alone â without curated sources â can produce a 5T-token dataset competitive with the mix-based Pile and RedPajama. Falcon 40B/180B were trained on it.
Common use cases
- Web-only pre-training
- Filtering research