GenAI interview guides / Speculative Decoding Interview Guide
Speculative Decoding Interview Guide
Explain how speculative decoding reduces inter-token latency and when draft/verification overhead is worth it.
Mechanism
A cheaper proposer predicts future tokens and the target model verifies them in fewer expensive steps.
When it helps
Best gains are typically decode-bound, medium-to-low-QPS, memory-bound workloads with good acceptance.
Trade-offs
Low acceptance, proposer overhead, memory, tokenizer mismatch, or heavy batching can erase the benefit.
Serving view
Measure accepted tokens per target step plus end-to-end p50/p95 latency.