AMD Instinct MI300: Detailed Architecture and Performance Analysis

AMD has published a detailed description of its Instinct MI300 accelerators. From an engineering standpoint, the MI300 builds on the ideas behind the MI200 but uses a significantly more complex CoWoS (Chip-on-Wafer-on-Substrate) packaging approach. The chip can contain up to eight accelerator modules (XCDs) and four input/output modules (IODs), connected via the Infinity Fabric interconnect. The total transistor count reaches 153 billion. The MI300 comes in two versions: the MI300X, a "pure" accelerator with eight XCDs and 192GB of HBM3 memory, and the MI300A, a hybrid featuring six XCDs and three CPU modules (24 Zen 4 cores) with 128GB of HBM3. The thermal design power ranges from 750 to 760 watts. The CDNA3 architecture has been completely redesigned: the matrix cores have been improved, cache bandwidth has been increased, and support for FP8 and TF32 data types as well as sparse data has been added. According to AMD, the MI300X outperforms NVIDIA's H100 SXM across all computing modes, especially in FP64, and offers almost 2.5 times more HBM3 memory, giving it an advantage in training and inference of large AI models. In HPC (High-Performance Computing) tasks, such as the OpenFOAM motorbike benchmark, the MI300A outperformed the H100 by a factor of four while consuming less TDP (thermal design power). A platform built on eight MI300X units does not require external switches, unlike NVIDIA's H100 setup. AMD also noted that it responded to NVIDIA's claims, stating that the MI300X is still faster than the H100 in inference workloads.