AI SSDs: A Storage Paradigm Shift in LLM Inference

AI infrastructure competition is shifting from raw GPU count and compute power to a more holistic approach where compute, network, memory, and storage work together per token. This is driven by long-context models, complex reasoning, and multi-agent systems, requiring efficient management of KV Cache and MoE expert weights. Two representative signals are Moonshot AI's Mooncake and NVIDIA's CMX. Mooncake is a KV Cache-centric disaggregated architecture that organizes idle CPU, DRAM, SSD, and NIC resources into a distributed cache pool, improving effective request capacity by 59% to 498% in tests. NVIDIA's CMX Context Memory Storage Platform adds a 'G3.5' layer between HBM and shared storage, providing PB-scale shared context memory within a GPU pod, potentially offering up to 5x improvement in sustained token throughput and power efficiency. These systems highlight that inference state is becoming a key resource, and storage must participate in placement, migration, and reuse. Consequently, AI SSDs are emerging, with two categories: AI-enhanced enterprise SSDs (e.g., InnoGrit Dongting-N3X, Huawei OceanDisk LC 560) and inference-participating AI SSDs. Three notable routes for the latter are Phison's aiDAPTIV (using dedicated cache SSDs and middleware), Longsys's SPU+iSA (leveraging storage-side intelligence), and Infplane-Maxio's cross-layer coordination. The competition in AI SSDs will ultimately be about the complete data path, requiring collaboration across NAND media, FTL, firmware, drivers, middleware, and inference frameworks. AI SSDs will not replace HBM, VRAM, or DRAM but will form a finer storage hierarchy, with data placed based on access temperature and migration managed by software and hardware.