Latency & Serving interview questions.
A curated set of 22 questions that repeatedly exercise Latency & Serving concepts in AI/ML interviews.
How should dropout behave at test time?ML Fundamentals · medium · Evidence 82/100Design a content moderation ML systemML System Design · hard · Evidence 77/100Design an ML system using traffic dataML System Design · hard · Evidence 77/100LRU Cache With Delete and LastPython & DSA · medium · Evidence 77/100How do you account for observed and unobserved confounding?ML Fundamentals · hard · Evidence 75/100How would you design an experiment when confounding is a concern?ML Fundamentals · hard · Evidence 75/100Design a LinkedIn Classification SystemML System Design · hard · Evidence 41/100How would you turn a research model into a production ML system?ML System Design · hard · Evidence 41/100Transformation + Sorting AI Coding ProblemML Coding & PyTorch · medium · Evidence 41/100How would you profile GPU inference latency end to end?ML System Design · hard · Evidence 41/100How do throughput and tail latency trade off in model serving?ML System Design · hard · Evidence 41/100How do continuous batching and paged attention improve LLM serving?GenAI & LLM · hard · Evidence 41/100How would you debug unexpectedly low GPU utilization during inference?ML System Design · hard · Evidence 41/100Explain the architecture of an LLM serving stack built around vLLM.GenAI & LLM · hard · Evidence 41/100When would you choose approximate nearest-neighbor search over exact search?ML Fundamentals · medium · Evidence 41/100How would you reduce memory movement in a custom CUDA inference kernel?ML Coding & PyTorch · hard · Evidence 41/100What bottlenecks would you optimize in a CUDA kernel used for model inference?ML Coding & PyTorch · hard · Evidence 41/100How would you handle a missing image or missing text modality at inference time?ML System Design · hard · Evidence 41/100How would you quantize a neural network for faster inference?GenAI & LLM · medium · Evidence 40/100Design concurrent LLM servingGenAI & LLM · hard · Evidence 37/100Design a Second-Pass Feed RankerML System Design · hard · Evidence 34/100What Is an LLM?GenAI & LLM · easy · Evidence 34/100