Top NLP Interview Questions: LLM Architectures, Inference and Optimization
Mistral
Google/DeepMind
This article provides a checklist of key topics and questions for technical interviews on modern LLM architectures and inference optimization. It covers architectural differences from vanilla Transformer (Pre-Norm, RMSNorm, SwiGLU, RoPE, GQA/MQA, MoE), explains why LLM inference is expensive, and describes batching, KV-cache, quantization, and other acceleration techniques.
Modern generative LLMs are mostly decoder-only transformers, with families like GPT, LLaMA, Mistral, Qwen, and Gemma differing in openness, parameter count (from hundreds of millions to hundreds of billions), architecture details (Dense vs MoE), attention type, normalization, positional encoding, MLP specifics, context window length (e.g., 4k, 32k, 128k tokens), multimodal support, training, and license. Compared to the classic Transformer, modern LLMs use Pre-Norm instead of Post-Norm (more stable training), RMSNorm instead of LayerNorm (cheaper), SwiGLU gated FFN (more expressive but adds a third linear layer), RoPE positional embeddings, MQA/GQA instead of full multi-head attention, sometimes MoE (mixture of experts) for conditional computation, longer context, and inference optimizations. Inference is expensive because autoregressive generation reads all weights per token, KV-cache grows with context and batch, and there are two phases: prefill (compute-bound) and decode (memory-bound). Latency (time to first token, inter-token latency) vs throughput trade-offs are key; optimization techniques include KV-cache, paged attention, continuous batching, speculative decoding, quantization, and distillation.
- Abbreviations
- GPT = Generative Pre-trained Transformer — генеративный предобученный трансформер
- LLM = Large Language Model — большая языковая модель
- FFN = Feed-Forward Network — сеть прямого распространения
- ReLU = Rectified Linear Unit — линейный выпрямитель
- GELU = Gaussian Error Linear Unit — гауссовская линейная единица ошибок
- MoE = Mixture of Experts — смесь экспертов
- RMSNorm = Root Mean Square Layer Normalization — нормализация по среднеквадратичному
- SwiGLU = Swish-Gated Linear Unit — gated линейный блок с функцией Swish
- RoPE = Rotary Position Embedding — вращательное позиционное кодирование
- MHA = Multi-Head Attention — многоголовое внимание
- MQA = Multi-Query Attention — многозапросное внимание
- GQA = Grouped Query Attention — группированное внимание по запросам
- MLP = Multilayer Perceptron — многослойный перцептрон
- GPU = Graphics Processing Unit — графический процессор
- API = Application Programming Interface — интерфейс программирования приложений
- KV = Key-Value — ключ-значение
- HBM = High Bandwidth Memory — память с высокой пропускной способностью
- TTFT = Time To First Token — время до первого токена
Source: Habr — хаб ИИ —
original