DeepSeek 0731 on DGX Spark: Overclocking to 68 tok/s and Why the Card Author Got 57
DeepSeek
NVIDIA
vLLM
A developer benchmarks DeepSeek-V4-Flash on a dual NVIDIA DGX Spark setup, initially measuring 42.5 tok/s versus the 57 tok/s claimed on the model card. After switching the vLLM runtime image and explicitly enabling the B12X MoE backend, they achieved 67.6 tok/s on the card-like profile, attributing the improvements to specific configuration changes.
The author deployed DeepSeek-V4-Flash on two DGX Spark nodes connected via QSFP 200G with RoCEv2, using tensor parallelism across nodes. Initial measurements showed 42.5 tok/s, about 25% below the 57 tok/s claimed on the model card. After testing several hypotheses (temperature, prefill accounting, context length), the main difference was attributed to the runtime image: switching from a vLLM 0.21.1-based build to one with vLLM 0.25.2 gave +22% on the same checkpoint, increasing output tokens per verification step from 3.27 to 4.80. Further gains came from explicitly setting --moe-backend flashinfer_b12x, since in the custom image B12X is excluded from auto-selection; this added +13.3% on average on a card-like profile (59.7 to 67.6 tok/s). The author also notes a new checkpoint release (0731) with improved agent benchmarks, and acknowledges parallel work by tonyd2wild on similar optimizations.
- Abbreviations
- DGX = Deep Learning GPU Accelerator — GPU-ускоритель для глубокого обучения
- GB10 = Grace Blackwell 10 — чип NVIDIA Grace Blackwell 10
- SM = Streaming Multiprocessor — потоковый мультипроцессор
- QSFP = Quad Small Form-factor Pluggable — квадруплексный малый форм-фактор
- RoCEv2 = RDMA over Converged Ethernet version 2 — RDMA поверх конвергентного Ethernet версии 2
- TP = Tensor Parallelism — тензорный параллелизм
- vLLM = Virtual Large Language Model — виртуальная большая языковая модель
- MoE = Mixture of Experts — смесь экспертов
- SSE = Server-Sent Events — события, отправляемые сервером
- NCCL = NVIDIA Collective Communications Library — библиотека коллективных коммуникаций NVIDIA
- FP4 = Floating Point 4-bit — числа с плавающей запятой 4-бит
- FP8 = Floating Point 8-bit — числа с плавающей запятой 8-бит
- KV = Key-Value — ключ-значение
- MLX = Metal Performance Shaders Graph — граф производительности Metal
- CLI = Command-Line Interface — интерфейс командной строки
- API = Application Programming Interface — интерфейс программирования приложений
- TTFT = Time To First Token — время до первого токена
- E2E = End-to-End — сквозной
- CBRN = Chemical, Biological, Radiological, Nuclear — химический, биологический, радиологический, ядерный
- RLHF = Reinforcement Learning from Human Feedback — обучение с подкреплением на основе человеческой обратной связи
Source: Habr — хаб ИИ —
original