GenAI interview guides / LLM Inference Optimization Interview Guide
LLM Inference Optimization Interview Guide
Optimize LLM serving across throughput, time-to-first-token, inter-token latency, memory, and cost.
Split latency
Separate queueing, tokenization, prefill, first-token latency, decode, tool time, and network time.
Serving levers
Continuous batching, prefix caching, quantization, optimized kernels, parallelism, and prefill/decode scheduling target different bottlenecks.
Capacity
Use prompt/output length distributions and SLOs, not only requests per second. Track KV pressure, batch size, and tail latency.
Quality guardrail
Every cost/latency optimization needs quality and numerical regression tests.