The 60-second answer
Explain Q, K, and V projections, scaled dot-product attention, masking, softmax, and the weighted sum of values with tensor shapes. For multi-head attention, show how heads split/project dimensions and are concatenated before the output projection.
Build the answer in this order
Explain Q, K, and V projections, scaled dot-product attention, masking, softmax, and the weighted sum of values with tensor shapes.
For multi-head attention, show how heads split/project dimensions and are concatenated before the output projection.
Connect architecture to positional information, residuals, normalization, feed-forward blocks, and the training objective.
Discuss O(n²) attention cost, KV caching for autoregressive decoding, numerical stability, and serving latency.
A useful interview mental model
This is the shape of a strong answer—not a script to memorize.
Senior-level signal
- Reason about memory bandwidth and KV-cache growth, not only FLOPs, when discussing real inference performance.
- Compare dense attention with sparse/windowed/linear alternatives only after stating the task and context-length trade-off.
What the interviewer is really testing
Likely follow-up questions
Common weak-answer patterns
- Jumping to a model before defining the product contract.
- Listing components without bottlenecks, metrics, or failure handling.
- Ignoring data quality, serving latency, monitoring, and iteration.