Comparison of Large Language Model Architectures: From DeepSeek V3 to OLMo 2

Sebastian Raschka presented an overview of architectural solutions in modern open large language models, focusing on text capabilities and leaving multimodality outside the scope of the article. In the section on DeepSeek V3/R1, two key components are analyzed in detail: Multi-Head Latent Attention (MLA) and Mixture-of-Experts (MoE). MLA compresses key and value tensors into a lower-dimensional space before storing them in the KV cache, reducing memory usage, and then reconstructs them during inference; according to ablation studies from the DeepSeek-V2 paper, MLA demonstrates better modeling quality than Grouped-Query Attention (GQA). MoE in DeepSeek V3 uses 256 experts per module, of which only 9 are active (one shared and eight selected by the router), allowing the model with 671 billion parameters to use only 37 billion per inference step. In the section on OLMo 2, it is noted that the model from the Allen Institute for AI is notable for its transparency and uses traditional Multi-Head Attention (MHA) instead of MLA or GQA. The key architectural difference of OLMo 2 is the placement of normalization layers: it applies Post-Norm (RMSNorm after attention and FeedForward sublayers), unlike the more common Pre-Norm, which, according to research, improves gradient convergence during initialization.