Latest innovations in LLM architectures: shared KV caches, per-layer embeddings, and compressed attention

The article discusses the latest architectural changes in open LLMs aimed at reducing costs when handling long context. Google's Gemma 4 models (E2B, E4B) use a shared KV cache: later layers reuse keys and values from previous layers of the same attention type, reducing the KV cache size by half (e.g., for E2B with a 128K context, this saves 2.7 GB at bfloat16). Additionally, the E2B and E4B variants employ per-layer embeddings (PLE): each transformer block receives a small token-specific vector from a shared 'packed' table, increasing model capacity without a proportional increase in the number of parameters in the main blocks. Poolside's Laguna XS.2 implements per-layer attention budgeting: out of 40 layers, only 10 have full (global) attention, while the remaining 30 use a sliding window (512 tokens), saving resources. Finally, DeepSeek V4 uses mHC (multi-head compression) and compressed attention mechanisms for further optimization. Although these techniques may seem like minor tweaks, they significantly reduce memory and computational load when processing long sequences.