Concept
Pre-training
Initial unsupervised training of a model on massive corpora to learn general patterns.
Definition
Pre-training is the expensive first stage — frontier runs cost tens of millions of dollars — where a model learns language, code, or vision from scratch on web-scale data. The resulting base model is then specialised via fine-tuning, instruction tuning, and RLHF. Compute and data scale predict downstream quality.
Common use cases
- Foundation model creation
- Domain pre-training
- Tokenizer co-training