Scaling LLM Training: From a Single Chip to a Data Center. Chapter 4. Training a Transformer
The chapter reviews four types of parallelism for training transformers on multiple accelerators: data parallelism, fully-sharded data parallelism (FSDP), tensor parallelism, and pipeline parallelism. It discusses their advantages, disadvantages, and when to combine them, emphasizing the importance of achieving computation-bound performance for scalability.
The article explains how to train a transformer efficiently and at scale. It introduces notation for model dimensions (D, F, B, T, L) and hardware (C, W, X, Y, Z). The four parallelism types are described: data parallelism splits the batch across accelerators, requiring AllReduce for gradient aggregation, and is effective when the model fits in one accelerator. FSDP (ZeRO) shards model weights, gradients, and optimizer states, using AllGather and ReduceScatter, and is computation-bound with small batch sizes. Tensor parallelism shards weight matrices, exchanging activations, and is best combined with FSDP to reduce activation size relative to weights. Pipeline parallelism splits the model across layers, minimizing data transfer but introducing 'pipeline bubbles,' which can be mitigated by microbatching or overlapping computations. The article notes that FSDP+TP combination is almost always computation-bound, and pipelining is common for GPU clusters. It concludes that the goal is to scale while managing overhead.
- Abbreviations
- FSDP = Fully-Sharded Data Parallelism — Полностью сегментированный параллелизм данных
- TP = Tensor Parallelism — Тензорный параллелизм
- MLP = Multi-Layer Perceptron — Многослойный перцептрон
- LLM = Large Language Model — Большая языковая модель
Source: Habr — хаб ИИ —
original