From GPT-2 to Kimi K3: Scale Grows 22,580x in Seven Years, Main Architecture Thread Builds a "Memory Operating System"

The article traces seven years of architecture evolution, analyzing technical changes behind the scale growth of large models and how Kimi K3 breaks through traditional Transformer limitations with new memory mechanisms. GPT-2 used a decoder-only architecture with token and position embeddings. KV Cache stores key-value vectors to avoid recomputation. Linear Attention applies feature maps (e.g., ELU+1) to q and k, compressing KV vectors into a fixed-size state matrix. DeltaNet extends Linear Attention with a delta rule to selectively update memory, reducing information interference. Gated DeltaNet combines Mamba's gating with the delta rule, adding a decay parameter α. Kimi Linear improves Gated DeltaNet with fine-grained channel-wise gating. Kimi K3's language backbone has 23 macro-loops, each containing three Kimi Delta Attention layers and one Multi-Head Latent Attention layer. It uses dense FFN in the first layer and Latent MoE in all others. Kimi K3 has 898 experts total; two shared experts process every token, and the router selects 16 experts per token from the remaining 896.