The 60-second answer
Break end-to-end latency into queueing, preprocessing, host/device transfer, kernel execution, synchronization, decoding, and post-processing before optimizing. Profile utilization, memory bandwidth, occupancy, batch shape, cache behavior, and p50/p95/p99 latency to identify whether the bottleneck is compute, memory, launch overhead, or scheduling.
Build the answer in this order
Break end-to-end latency into queueing, preprocessing, host/device transfer, kernel execution, synchronization, decoding, and post-processing before optimizing.
Profile utilization, memory bandwidth, occupancy, batch shape, cache behavior, and p50/p95/p99 latency to identify whether the bottleneck is compute, memory, launch overhead, or scheduling.
Apply batching, kernel fusion, mixed precision, caching, layout changes, or paged/continuous-batching techniques only when profiling supports them.
Validate under realistic concurrency and sequence-length distributions while tracking throughput, tail latency, memory headroom, and failure rate.
A useful interview mental model
This is the shape of a strong answer—not a script to memorize.
Senior-level signal
- Design admission control and backpressure so throughput gains do not create OOMs or catastrophic tail latency.
- Tie kernel-level wins to end-to-end request performance; microbenchmarks alone are not a production success metric.
What the interviewer is really testing
Likely follow-up questions
Common weak-answer patterns
- Ignoring shape, dtype, device, masking, or broadcasting assumptions.
- Using a framework call without explaining the underlying operation.
- Skipping gradient, numerical-stability, and batching checks.