Gigatoken: BPE Tokenizer in Rust Encodes Text at 24.53 GB/s, Up to 989x Faster Than HuggingFace Tokenizers

Stanford PhD student Marcel Roed released Gigatoken, a BPE tokenizer in Rust with Python bindings, achieving speeds of up to 24.53 GB/s on a 144-core AMD EPYC, 989x faster than HuggingFace Tokenizers and 681x faster than tiktoken. The speedup comes from a handwritten SWAR pre-tokenizer and pre-token caching.

OpenAI Hugging Face Google/DeepMind Meta Alibaba/Qwen DeepSeek Moonshot AI NVIDIA Microsoft Allen Institute for AI Answer.AI Mistral