Concept
Tokenization
Process of splitting text into the discrete units (tokens) that a model processes.
Definition
Tokenization breaks raw text into a model's vocabulary entries â usually subwords via Byte-Pair Encoding or SentencePiece. Token counts determine context-window usage and API billing. The same string can tokenise very differently across providers, which is why exact counts must always be measured.
Common use cases
- Cost estimation
- Context budgeting
- Tokenizer training