Framework & Tool
tiktoken
OpenAI's fast Rust+Python BPE tokeniser library, the reference for counting GPT tokens.
Definition
tiktoken is OpenAI's official tokeniser, distributed with both Python bindings and a Rust core. It is the authoritative way to count tokens billed by the OpenAI API; many other libraries copy its encodings for compatibility.
Common use cases
- Token counting
- Cost estimation
- Context budgeting