Inferência de LLMs: Do KV-Cache à Implantação em Produção

Sasha Ryzhov, MLOps engineer at hh.ru, shares his experience deploying LLMs on on-premise hardware. He explains that inference efficiency depends primarily on video memory management, not on the model itself. The KV-cache is analyzed in detail, which shifts the decode stage from compute-bound to memory-bound. Two main inference frameworks are considered: vLLM with its PagedAttention mechanism, which solves the problems of internal and external fragmentation, and SGLang with RadixAttention, which implements more flexible prefix caching. Additional SGLang features are also described, such as Compressed FSM for function calling, cache-aware scheduling, and dp-attention. The evolution of attention mechanisms is mentioned: from MHA through GQA and MLA to DSA — each successive step reduces the size of the KV-cache. Practical recommendations are given: for H100 and newer, choose SGLang; for older GPUs, use vLLM; the optimal model size is MoE up to 120B or Dense up to 32B; for low latency, use Tensor Parallelism; for high throughput, use Data Parallelism; splitting prefill and decode provides maximum throughput but requires more engineering resources. Speculative decoding is not recommended for high-load systems, as it consumes additional memory for the draft model and reduces overall throughput.