New DeepSeek-V3 technical report: how hardware-software co-design enables low-cost training of large models
DeepSeek
NVIDIA
Alibaba/Qwen
Meta
A new technical paper from the DeepSeek team, with CEO Wenfeng Liang as co-author, explores hardware-aware model co-design to reduce LLM training costs. Using DeepSeek-V3 trained on 2048 NVIDIA H800 GPUs as a case study, the paper details innovations in memory efficiency (MLA), sparse computation (DeepSeekMoE), FP8 training, and interconnect-aware routing.
The 14-page paper, 'Scaling Challenges and Reflections on Hardware for AI Architectures,' analyzes how hardware constraints (memory, compute, interconnect bandwidth) shape LLM design. DeepSeek-V3 uses Multi-head Latent Attention (MLA) to compress KV cache to 70 KB per token vs. 516 KB for LLaMA-3.1 405B. Its DeepSeekMoE architecture activates only 37B of 671B parameters per token, reducing per-token FLOPs to ~250 GFLOPS vs. 394 for Qwen2.5-72B and 2448 for LLaMA-3.1 405B. The model employs FP8 mixed-precision training and LogFMT lower-precision communication, cutting communication volume by 50% over BF16. To mitigate NVIDIA H800's reduced NVLink bandwidth (400 GB/s vs. 900 GB/s in H100), node-aware routing groups 256 experts into 8 node-local groups, limiting each token to at most 4 nodes. The paper advocates for unified scale-up/scale-out interconnect and dedicated communication co-processors.
- Сокращения
- LLM = Large Language Model — большая языковая модель
- MLA = Multi-head Latent Attention — многоголовочное латентное внимание
- MoE = Mixture of Experts — смесь экспертов
- FP8 = 8-bit Floating Point — 8-битная плавающая точка
- BF16 = Brain Floating Point 16-bit — 16-битный мозговой формат с плавающей точкой
- KV = Key-Value — ключ-значение
- HBM = High Bandwidth Memory — высокопропускная память
- TP = Tensor Parallelism — тензорный параллелизм
- PP = Pipeline Parallelism — конвейерный параллелизм
- EP = Expert Parallelism — экспертный параллелизм
- NVLink = NVIDIA NVLink — NVIDIA NVLink
- IB = InfiniBand — InfiniBand
- GPU = Graphics Processing Unit — графический процессор
- SM = Streaming Multiprocessor — потоковый мультипроцессор
- PCIe = Peripheral Component Interconnect Express — шина PCI Express
- NIC = Network Interface Card — сетевая интерфейсная карта
- UEC = Ultra Ethernet Consortium — Ultra Ethernet Consortium
- UALink = Ultra Accelerator Link — Ultra Accelerator Link
- MPFT = Multi-Plane Fat-Tree — Multi-Plane Fat-Tree
- GQA = Grouped Query Attention — группированное внимание запросов
- MQA = Multi-Query Attention — многозапросное внимание
- TPS = Tokens Per Second — токенов в секунду
Source: Synced —
original