LLM Inference: From KV-Cache to Production Deployment

Sasha Ryzhov, an MLOps engineer at hh.ru, shares experience deploying LLMs on-premise hardware. He explains that inference efficiency primarily depends on GPU memory management, not the model itself. The KV cache is discussed in detail, which transitions the decode stage from compute-bound to memory-bound. Two main inference frameworks are considered: vLLM with the PagedAttention mechanism, solving internal and external fragmentation issues, and SGLang with RadixAttention, implementing more flexible prefix caching. Additional SGLang features are described, such as Compressed FSM for function calling, cache-aware scheduling, and dp-attention. The evolution of attention mechanisms is mentioned: from MHA through GQA and MLA to DSA, with each step reducing KV cache size. Practical recommendations are given: for H100 and newer, choose SGLang; for older GPUs, choose vLLM; optimal model size: MoE up to 120B or Dense up to 32B; for low latency, use Tensor Parallelism; for high throughput, use Data Parallelism; separating prefill and decode yields maximum throughput but requires more engineering resources. Speculative decoding is not recommended for high-load systems because it consumes extra memory for the draft model and reduces overall throughput.