The 60-second answer
Explain token embeddings plus positional information feeding repeated attention and feed-forward blocks. Self-attention forms Q, K, and V, computes scaled similarities, then mixes value vectors.
Build the answer in this order
1
State tensor contract
Explain token embeddings plus positional information feeding repeated attention and feed-forward blocks.
2
Implement the mechanism
Self-attention forms Q, K, and V, computes scaled similarities, then mixes value vectors.
3
Check numerics + gradients
Multi-head attention learns multiple interaction subspaces while residuals and normalization stabilize training.
4
Test shapes and edge cases
Call out quadratic attention cost with sequence length and masking requirements.
A useful interview mental model
This is the shape of a strong answer—not a script to memorize.
01Shapes
02Forward pass
03Loss / grads
04Numerics
05Tests
Senior-level signal
- Discuss KV caching and efficient attention kernels for serving.
- Compare encoder-only, decoder-only, and encoder-decoder objectives based on the task.
What the interviewer is really testing
Tensor fluency, shape reasoning, numerics, gradients, batching, device awareness, and the ability to debug—not API memorization.
Likely follow-up questions
What are the tensor shapes at each step?
Where could numerical instability or silent broadcasting appear?
How would you verify gradients and batched behavior?
Common weak-answer patterns
- Ignoring shape, dtype, device, masking, or broadcasting assumptions.
- Using a framework call without explaining the underlying operation.
- Skipping gradient, numerical-stability, and batching checks.