LLM Architecture Innovations: KV Sharing, Per-Layer Embeddings, and Compressed Attention

Google/DeepMind

Poolside

DeepSeek

Technology Innovation Institute

Sebastian Raschka reviews recent open-weight LLM architecture advances focusing on long-context efficiency. Key techniques include KV sharing across layers (Gemma 4), per-layer embeddings (Gemma 4 E2B/E4B), layer-wise attention budgeting (Laguna XS.2), compressed convolutional attention (ZAYA1), and mHC with compressed attention (DeepSeek V4). These reduce KV cache size and memory traffic for reasoning and agent workflows.

Sebastian Raschka surveys recent open-weight LLM architecture changes aimed at reducing long-context costs. In Gemma 4 E2B/E4B, later layers reuse key-value states from earlier layers (cross-layer attention), saving about half the KV cache size (e.g., 2.7 GB at 128K context). Additionally, these models use per-layer embeddings (PLE) to increase capacity without expanding the main transformer stack—the 'effective' parameter count is smaller than the total including embeddings. Laguna XS.2 by Poolside varies attention cost by layer, with 30 sliding-window (window 512) and 10 full-attention layers. ZAYA1-8B introduces compressed convolutional attention, and DeepSeek V4 employs mHC and compressed attention. These tweaks are motivated by the growing need for efficient long-context processing in reasoning models and agent workflows.

Сокращения: KV = Key-Value — ключ-значение; LLM = Large Language Model — большая языковая модель; GQA = Grouped Query Attention — группированное внимание по запросам; MQA = Multi-Query Attention — множественное внимание по запросам; MoE = Mixture of Experts — смесь экспертов; PLE = Per-Layer Embeddings — послойные эмбеддинги

Source: Sebastian Raschka — original