Token Cramming: Perfect Text Compression Does Not Preserve Meaning – AIRI Study
Researchers from AIRI (FusionBrain) identified critical issues with the token cramming method, which allows packing thousands of tokens into a single embedding. It turned out that even at 99.96% reconstruction accuracy, the model loses its ability to reason over compressed text. A new protocol called progressive cramming was proposed, guaranteeing 100% reconstruction but showing that text recovery does not equal understanding.
Meta
EleutherAI
Hugging Face
Google/DeepMind
Alibaba/Qwen
Habr — хаб ИИ27.07 · 10:03