Concept
aka BPE
Byte-Pair Encoding
Tokenisation algorithm that iteratively merges the most-frequent adjacent token pairs.
Definition
BPE builds a vocabulary by starting from individual bytes and greedily merging the most frequent adjacent pair until a target vocab size is reached. Variants (byte-level BPE, SentencePiece BPE) underlie nearly every modern LLM tokeniser.
Common use cases
- Tokeniser training
- Multilingual support
- Open-vocabulary modelling