Comparison of Large Language Model Architectures: From DeepSeek V3 to OLMo 2

In his article, Sebastian Raschka examines the architectural features of modern large language models, comparing DeepSeek V3/R1, OLMo 2, and others. DeepSeek V3 uses Multi-Head Latent Attention (MLA), which compresses keys and values into a low-dimensional space to save KV cache memory, and Mixture-of-Experts (MoE) with 256 experts, of which only 9 are active (one shared expert and 8 selected by the router), allowing it to use only 37 billion parameters per step despite having 671 billion total. OLMo 2 from the Allen Institute for AI, on the other hand, employs traditional Multi-Head Attention but with Post-Norm (RMSNorm after attention and feed-forward network layers) and QK-norm to stabilize training. The article also mentions Grouped-Query Attention as an alternative to Multi-Head Attention used in other models.