Concept
Vocabulary
Fixed set of tokens a tokenizer and model can represent, typically 32k to 200k entries.
Definition
The vocabulary is the discrete alphabet the model emits and consumes. Bigger vocabularies compress text into fewer tokens (good for cost and context) but blow up the embedding matrix. Tokeniser+vocab choice is locked in at pre-training and cannot be swapped post-hoc.
Common use cases
- Multi-lingual support
- Domain tokenisers
- Cost reduction