Mini-PC on Strix Halo under parallel load: 236 tok/s on 32 concurrent requests and three errors
Google/DeepMind
A Beelink GTR9 Pro with Ryzen AI Max+ 395 achieved 236 tok/s aggregate on 32 concurrent requests in short runs, sustaining 226 tok/s average over 30 minutes without throttling. The author discovered a reproducible throughput drop between 8 and 10 concurrent clients across all tested models, and found that speculative decoding (MTP) accelerated single-client performance by 42% but became a penalty under concurrency.
Testing a Beelink GTR9 Pro (Ryzen AI Max+ 395, Radeon 8060S, 128 GB unified memory) with llama.cpp via Vulkan/RADV, the author ran parallel inference benchmarks. Qwen 3.6 35B-A3B in Q4_K_M showed a throughput valley from 149 tok/s at 8 clients down to 99 at 10, recovering to 160 at 32 clients. This valley was reproducible across all tested models (Qwen in three quants and Gemma 4 26B in QAT) and could not be attributed to a single model or quant. The best aggregate was 236 tok/s with Gemma 4 at 32 clients (median of 75-second runs). A 30-minute endurance test averaged 226.4 tok/s with stable temperatures (78°C) and power (113 W), with 1.4% errors from missing server timings. Speculative decoding (MTP) boosted single-client speed by 42% (53.3→75.6 tok/s) but hurt concurrency: at 4 clients it regressed 31%, at 32 clients it lost 33% throughput. The author notes three mistakes caught before publication by control measurements.
- Сокращения
- MTP = Multi-Token Prediction — Мультитокеновое предсказание
- MoE = Mixture of Experts — Смесь экспертов
- QAT = Quantization-Aware Training — Обучение с учётом квантования
- KV = Key-Value — Ключ-значение
- RADV = Radeon Vulkan Driver — Драйвер Radeon Vulkan
Source: Habr — хаб NLP —
original