tokenizers v1: encode, decode and scaling, measured
2026-09-21 · Hugging Face
tokenizers v1: encode, decode and scaling, measured
Background and Motivation
Historically, tokenizers have not been the bottleneck in machine learning workflows, as tokenization is computationally light compared to the heavy modeling in the rest of the pipeline. However, as models become faster and workloads scale—such as training on massive datasets, serving concurrent requests, or processing long inputs—the tokenizer can put enough pressure on the system to starve the model of data.
To address this, the upcoming version 1 of tokenizers focuses heavily on performance. Tokenization should be lightweight and scale with your workflow, ensuring GPUs never sit idle waiting for the CPU to finish tokenization. Compared to v0.23, v1 is often tens of times faster.
Acknowledgments and Ecosystem
This refactor was made possible by the open-source ecosystem. Libraries like gigatoken, tiktoken, kitoken, tokie, fastokens, wordchipper, and ai-tokenizer have pushed the boundaries of fast tokenization. The team also thanked IBM, NVIDIA, and the ExecuTorch team for contributing patches and helping test across a wide range of hardware.
What V1 Is
Version 1 produces the exact same token IDs as v0.23. The goal was to preserve the output, API, vocabulary, and merge ranks while improving everything else. The library remains general across tokenizer families rather than specializing solely in BPE, meaning v1 loads everything v0.23 loaded.
The tokenization pipeline consists of four stages:
1. Normalization: Applies operations like lowercasing or Unicode normalization to raw text.
2. Pre-tokenization: Splits text into smaller pieces called pre-tokens.
3. Model: Turns each pre-token into tokens and maps them to vocabulary IDs.
4. Post-processing: Adds any special tokens expected by the model.
Key Technical Improvements
Various changes were applied across the pipeline stages:
- Workspace split: The single crate became a workspace. `tk-encode` is the required runtime, while `tk-serialize`, `tk-convert`, and `tk-train` are linked only when needed.
- No-alloc model: The merge working set lives in a caller-owned scratch buffer, meaning the loop never touches the allocator.
- Bitcannon: The split pattern becomes Boolean operations over bitstreams, using SIMD instructions to find splits instead of a regex engine.
- Merge-loop rewrite: The pieces being merged form an intrusive doubly-linked list inside one preallocated buffer, so a merge updates two indices instead of moving data.
- Word cache: A thread-local memo from pre-token bytes to finished IDs ensures a repeated word is merged only once.
- Native parallelism: A single shared tokenizer encodes from many threads at once. Each thread draws its scratch buffer and word cache from its own sub-pool, eliminating queueing on a single lock.
The Split: Bitstreams Instead Of A Regex
BPE models use a regular expression to split input text into pre-tokens. Since merges happen inside a pre-token and never cross boundaries, this split dictates what the rest of the pipeline sees.
Because the regex is a fixed parameter that ships with the tokenizer, a general-purpose regex engine is unnecessary at runtime. Bitcannon uses a hand-written function leveraging SIMD (Single Instruction, Multiple Data) instructions to process many bytes at once. It views input bytes as parallel streams of bits, allowing boundaries to fall out of Boolean operations across whole registers rather than scanning one character at a time. It processes 64 bytes per register operation, an idea similar to Parabix and simdjson.
If the tokenizer's pattern is among the known grammars, this optimization applies; otherwise, it falls back to the regex path.