The 60-second answer
Learned Q and K project tokens into a matching space; V projects the content that is aggregated after matching. Scores come from QKᵀ/√d_k, masks alter allowed interactions, and softmax converts scores into mixing weights over V.
Build the answer in this order
Learned Q and K project tokens into a matching space; V projects the content that is aggregated after matching.
Scores come from QKᵀ/√d_k, masks alter allowed interactions, and softmax converts scores into mixing weights over V.
Multiple heads use separate projections so different relational patterns can be modeled in parallel before concatenation/output projection.
Track shapes carefully and explain that heads split representational width; they do not each see an independent copy of full hidden dimension by default.
A useful interview mental model
This is the shape of a strong answer—not a script to memorize.
Senior-level signal
- Senior answers discuss grouped-query/multi-query attention as serving optimizations that reduce KV-cache memory.
- Connect head dimension and scaling to numerical stability and hardware layout.
What the interviewer is really testing
Likely follow-up questions
Common weak-answer patterns
- Reciting a definition without mechanism or assumptions.
- Claiming one technique is always better without a data regime.
- Stopping before failure modes, validation, or deployment implications.