MoE on RTX 5090 Falls Short by 1.5x, and It's Not About Routing
A detailed benchmark on an RTX 5090 reveals that a Mixture-of-Experts model (Qwen3.5-35B-A3B) utilizes only 41% of the GPU's memory bandwidth, while a dense model (Qwen3.5-9B) achieves 72%. The root cause is insufficient bytes read per kernel launch, not expert routing. The author predicts performance for another model and across context depths with high accuracy.
The author benchmarks RTX 5090 using llama.cpp b10326 with Q4_0 quantization. The MoE model Qwen3.5-35B-A3B achieves 317.45 tokens/s, utilizing only 41% of memory bandwidth, while the dense Qwen3.5-9B achieves 239.62 tokens/s, utilizing 72%. They rule out CUDA graphs, routing, and GPU underutilization. Instead, they demonstrate via a synthetic ggml kernel that memory bandwidth scales with bytes read per kernel: small reads (a few MB) only achieve 20-50% of peak bandwidth. The MoE model's many expert projections result in many small kernels (average 4.6 MB per kernel), falling below the threshold for full bandwidth. The dense model reads larger matrices (22.2 MB per kernel), achieving 86% of peak. The author predicts the dense model's speed within 3.7% and context depth scaling within 2.4%, validating their model. They also note a tentative scaling law: the threshold for 90% bandwidth scales with GPU width (8x from RTX 4060 to 5090).
- Abbreviations
- MoE = Mixture of Experts — смесь экспертов
- GGUF = GPT-Generated Unified Format — формат файла модели для llama.cpp
- KV = Key-Value cache — кэш ключей и значений
- SM = Streaming Multiprocessor — потоковый мультипроцессор
Source: Habr — хаб ИИ —
original