Digest · past week

1. Gigatoken: BPE Tokenizer in Rust Encodes Text at 24.53 GB/s, Up to 989x Faster Than HuggingFace Tokenizers
   Gigatoken, a BPE tokenizer written in Rust by Stanford PhD student Marcel Rød, achieves text encoding speeds of 24.53 GB/s on a 144-core AMD EPYC system, outperforming HuggingFace tokenizers by 989x and OpenAI's tiktoken by 681x. The library supports 23 tokenizer families and attributes its speed to a hand-optimized pretokenizer using SWAR SIMD techniques and pretoken caching.