Hybrid Transformers Under Load: Where Gated DeltaNet Loses to Full Attention
An engineer tested two Qwen models, one with full attention and one with a hybrid architecture using Gated DeltaNet, under agentic load on an RTX 3090. The hybrid model wins on long context and high concurrency but loses on short dialogues with shared system prompts. The main issue is the cache block size jumping to 528 tokens, reducing prefix cache hit rates and forcing recomputation of a third of the prompt.
The hybrid model Qwen3.5-4B uses full attention in only 8 of 32 layers, with 24 recurrent Gated DeltaNet layers that keep a fixed-size state instead of growing KV cache. Under synthetic agentic load on a single RTX 3090 with vLLM 0.26.0, the hybrid showed 84% prefix cache hits versus 94% for the full attention Qwen3-4B, and recomputed three times more prompt tokens. The root cause is that recurrent state pages and attention KV pages share a common pool, forcing attention page size to match the state page size, raising the block size from 16 to 528 tokens. This reduces granularity of prefix matching and increases recomputation. The hybrid does have 3.2x larger effective cache capacity (297,720 tokens vs 93,408) but at a cost of 2.6 GiB less memory for cache and 2.3x larger activations. The larger block also increases per-token memory cost for state snapshots when full snapshotting is enabled.
- Сокращения
- KV = Key-Value
- TTFT = Time to First Token
- ITL = Inter-Token Latency
- GQA = Grouped Query Attention
- bf16 = bfloat16
- Mamba = Mamba (state space model name)
- SaaS = Software as a Service
- PCIe = Peripheral Component Interconnect Express
- CUDA = Compute Unified Device Architecture
- CPU = Central Processing Unit
- GPU = Graphics Processing Unit
- RAM = Random Access Memory
- GB = Gigabyte
- GiB = Gibibyte
- vLLM = vLLM (high-throughput serving engine)
Source: Habr — хаб ИИ —
original