New DeepSeek-V3 technical report: how hardware-software co-design enables low-cost training of large models

A new technical paper from the DeepSeek team, with CEO Wenfeng Liang as co-author, explores hardware-aware model co-design to reduce LLM training costs. Using DeepSeek-V3 trained on 2048 NVIDIA H800 GPUs as a case study, the paper details innovations in memory efficiency (MLA), sparse computation (DeepSeekMoE), FP8 training, and interconnect-aware routing.

DeepSeek NVIDIA Alibaba/Qwen Meta

NVIDIA Blackwell Ultra: 15 Pflops NVFP4, 288 GB HBM3e, no FP64

NVIDIA unveiled the Blackwell Ultra accelerator, featuring 208 billion transistors, 15 Pflops in NVFP4, 288 GB HBM3e memory, and new fifth-gen Tensor Cores. The chip omits FP64 but introduces NVFP4 format for efficient AI inference.

NVIDIA Taiwan Semiconductor Manufacturing Company