EleutherAI

Token Cramming: Perfect Text Compression Does Not Preserve Meaning – AIRI Study

Researchers from AIRI (FusionBrain) identified critical issues with the token cramming method, which allows packing thousands of tokens into a single embedding. It turned out that even at 99.96% reconstruction accuracy, the model loses its ability to reason over compressed text. A new protocol called progressive cramming was proposed, guaranteeing 100% reconstruction but showing that text recovery does not equal understanding.

Meta EleutherAI Hugging Face Google/DeepMind Alibaba/Qwen