GenAI interview guides / KV Cache Interview Guide
KV Cache Interview Guide
Explain KV caching, memory/latency trade-offs, long-context pressure, offloading, quantization, and prefix reuse.
Why it exists
Caching prior key/value tensors avoids recomputing them on every autoregressive decode step.
Memory trade-off
KV memory grows with context, concurrency, layers, heads, head dimension, and dtype; cache capacity can become the serving bottleneck.
Cache strategies
Dynamic, static, offloaded, quantized, and prefix-reused caches trade memory, bandwidth, latency, and implementation complexity.
Interview edge
Connect KV cache to prefill vs decode, continuous batching, eviction, isolation, and capacity planning.