The KV Cache Survival Guide: Why Your GPU Runs Out of Memory with Local LLMs
Meta
The article explains why GPUs run out of memory when running local LLMs, focusing on the KV cache as the culprit. It offers a survival guide with techniques to manage memory usage.
The article explains that running local large language models (LLMs) can cause GPUs to run out of memory, with the key-value (KV) cache being a major contributor. It describes how the KV cache stores intermediate attention values, growing linearly with sequence length and batch size, thus consuming significant memory. The guide provides strategies such as quantization, using flash attention, and going for smaller models or using CPU offloading. It emphasizes the trade-offs between memory savings and speed or quality.
- Сокращения
- GPU = Graphics Processing Unit — графический процессор
- LLM = Large Language Model — большая языковая модель
Source: Meta AI (GNews) —
original