Framework & Tool
SentencePiece
Google's language-agnostic tokeniser library implementing BPE and unigram models.
Definition
SentencePiece treats input text as a raw byte sequence and trains BPE or unigram tokenisers without language-specific pre-tokenisation. It powers Llama, T5, mBART, ALBERT and is the standard choice for multilingual models.
Common use cases
- Multilingual tokenisation
- Open-vocab modelling
- Pre-training pipelines