The 60-second answer
Separate projection/MLP compute from attention-matrix compute instead of quoting only O(n²). Q/K/V and output projections scale roughly with sequence length times hidden-size squared; attention score/value mixing scales with sequence-length squared times hidden size.
Build the answer in this order
Separate projection/MLP compute from attention-matrix compute instead of quoting only O(n²).
Q/K/V and output projections scale roughly with sequence length times hidden-size squared; attention score/value mixing scales with sequence-length squared times hidden size.
The feed-forward block is often a large linear-compute contributor because its intermediate dimension is several times hidden size.
State assumptions—batch size, number of layers/heads, forward vs training—and identify which term dominates for the regime given.
A useful interview mental model
This is the shape of a strong answer—not a script to memorize.
Senior-level signal
- Senior answers connect FLOPs to memory bandwidth and KV-cache behavior; theoretical FLOPs alone do not predict latency.
- Discuss how longer context changes the bottleneck differently during prefill versus autoregressive decode.
What the interviewer is really testing
Likely follow-up questions
Common weak-answer patterns
- Reciting a definition without mechanism or assumptions.
- Claiming one technique is always better without a data regime.
- Stopping before failure modes, validation, or deployment implications.