Token Cramming: Perfect Text Compression Does Not Preserve Meaning – AIRI Study

A group of researchers from the FusionBrain AIRI laboratory led by Dmitry Tarasov presented a paper at ICML 2026 that critically analyzes the method of token cramming—compressing hundreds or thousands of text tokens into a single input embedding of a frozen LLM. Previous work by colleague Yuri Kuratov (AIRI) showed that up to 1568 tokens can be "packed" into one vector with a reconstruction accuracy of 99.96% under teacher forcing. However, the new analysis revealed that errors concentrate on the first positions, which is fatal for autoregressive generation: with greedy decoding, accuracy drops to 40% for Llama-3.1-8B. The authors proposed a method called progressive cramming, which increases the prefix length by one token at a time, guaranteeing 100% reconstruction at each step. Using this method, the capacity of different models was measured: Llama-3.1-8B compresses an average of 1438 tokens, Pythia-1.4B—430, SmolLM2-1.7B—335, and Gemma-3-4B—214. Applying low-dimensional projections increases capacity by a factor of 1.2–3.3. Studying optimization trajectories showed that the optimizer's path in the embedding space lies in a low-dimensional subspace (30–100 components), although the solution set is high-dimensional. The main result: perfect reconstruction does not guarantee preservation of meaning. Adding a compressed embedding to the context reduces accuracy on HellaSwag and ARC-Easy (e.g., for Llama-3.1-8B from 61.9% to 40.8% on HellaSwag), and on 5-shot MMLU accuracy drops to nearly zero. The reason lies in the early layers of the transformer: damping attention to the embedding in the early layers restores reasoning ability. Compression capacity increases with model depth and width; for SmolLM2, the first 8 layers compress more than the full model.