KV Cache in LLMs: Understanding and Implementation from Scratch

KV cache stores key (K) and value (V) vectors from previous attention computations to reuse them when generating subsequent tokens, which speeds up inference by avoiding recomputation. Without a KV cache, each generation step recomputes keys and values for all previous tokens; with the cache, only the new token's vectors are computed and appended to the cache. The article provides a code implementation based on a GPT-like model from the author's book. The key changes are: adding cache buffers (cache_k and cache_v) in the MultiHeadAttention class, modifying the forward method to conditionally use the cache, adding a reset method, and propagating the use_cache flag through the model. In generation, when use_cache is True, the model processes only the new token after the initial prompt, while without cache it processes the full sequence each step. A simple performance comparison shows that the KV cache roughly doubles the generation speed for the tested example.