Making Knowledge Distillation Cheap Enough to Run at Scale
Multiverse Computing
NVIDIA
Meta
A new paper from Multiverse Computing introduces efficient offline knowledge distillation with cached top-K logits and a fused chunked KL loss, reducing VRAM usage by over 15x and enabling long-context distillation on a single GPU. The techniques cut training costs significantly, making large-scale distillation experiments practical.
Knowledge distillation, where a smaller student model learns to match a larger teacher model, is a standard technique for compressing large language models. Recent open-source LLMs like Kimi-K3 with 2.8 trillion parameters require about 3TB of VRAM, making compression essential. The paper 'Efficient Knowledge Distillation for LLMs: Offline Top-K Logits and a Fused Chunked KL Loss' addresses the high costs of distillation by caching the teacher's top-K logits offline, eliminating the need to keep the teacher in memory. It also introduces a fused chunked KL loss that computes the loss in chunks, avoiding the full vocabulary×sequence matrix. This reduces peak VRAM from 250GB to 128GB on a single H200, enabling distillation at 32K context on one GPU instead of four nodes, with 5x faster step times. The student model, distilled from Llama 3.1 8B Instruct to 3.2B parameters, retains most accuracy on benchmarks like BoolQ and HellaSwag and stays within nine points on MMLU.
- Abbreviations
- KL = Kullback-Leibler — Кульбака-Лейблера
- VRAM = Video RAM — видеопамять
- GPU = Graphics Processing Unit — графический процессор
- TFLOP = Tera Floating Point Operations per second — терафлопс
- MMLU = Massive Multitask Language Understanding — массовое понимание языка
- LLM = Large Language Model — большая языковая модель
Source: Hugging Face blog —
original