A Visual Guide to Attention Variants in Modern LLMs
OpenAI
Allen Institute for AI
Meta
Alibaba/Qwen
Google/DeepMind
Mistral
Hugging Face
Cohere
This article provides a visual overview of attention mechanisms in modern large language models, covering from standard multi-head attention (MHA) to grouped-query attention (GQA), multi-head latent attention (MLA), sparse attention, and hybrid architectures. It is accompanied by an LLM architecture gallery with 45 entries and poster versions.
Sebastian Raschka published a visual guide to attention variants in modern LLMs, alongside a gallery of 45 LLM architectures. The article covers multi-head attention (MHA), its historical context, and grouped-query attention (GQA) which reduces KV-cache costs by sharing key-value heads. GQA is a popular trade-off between efficiency and quality. Multi-head latent attention (MLA), used in DeepSeek V3, compresses the cache more aggressively while maintaining better performance. The article also discusses sparse attention and hybrid architectures, and includes example models for each variant. A poster version is available via Redbubble.
- Сокращения
- MHA = Multi-Head Attention — многоголовочное внимание
- GQA = Grouped-Query Attention — групповое внимание по запросам
- MLA = Multi-Head Latent Attention — многоголовочное латентное внимание
- KV = Key-Value — ключ-значение
- RNN = Recurrent Neural Network — рекуррентная нейронная сеть
- LLM = Large Language Model — большая языковая модель
Source: Sebastian Raschka —
original