NVIDIA Blackwell Ultra: 15 Pflops NVFP4, 288 GB HBM3e, no FP64

NVIDIA detailed the Blackwell Ultra accelerator, an enhanced version of the Blackwell architecture. The chip comprises two dies with 208 billion transistors on TSMC 4NP process, connected via NV-HBI with 10 TB/s bandwidth. It includes 160 streaming multiprocessors with 640 fifth-gen Tensor Cores delivering 15 Pflops in NVFP4 (without sparsity) and a unified L2 cache. Each SM has 128 CUDA cores, 4 Tensor Cores with second-gen Transformer Engine, 256 KB of Tensor Memory (TMEM), and special function units. The new NVFP4 format uses micro-block FP8 scaling and tensor-level FP32 scaling, achieving accuracy close to FP8 with 1.8x less memory. Blackwell Ultra boosts attention layer performance by doubling SFU throughput for softmax, reducing time to first token. Memory is 288 GB HBM3e (8 TB/s bandwidth), a 50% increase over Blackwell, enabling large models with 300+ billion parameters. External connectivity includes NVLink 5 (1.8 TB/s) and PCIe 6.0 x16. The Grace Blackwell Ultra superchip combines one Grace CPU with two Blackwell Ultras, offering up to 40 Pflops NVFP4 with sparsity and 1 TB unified memory. The GB300 NVL72 rack system integrates 36 superchips for 1.1 Exaflops FP4.