CrackML by @ml.with.umang
Interview questions / GenAI & LLM
GenAI & LLM interview question

Explain Transformers in an NLP context.

Explain Transformers in an NLP context. Structure your response as you would in a top-tier ML/AI engineering interview.

mediumconceptEvidence 75/1001 source reportTikTok

The 60-second answer

Explain Q, K, and V projections, scaled dot-product attention, masking, softmax, and the weighted sum of values with tensor shapes. For multi-head attention, show how heads split/project dimensions and are concatenated before the output projection.

Build the answer in this order

1
Define the mechanism

Explain Q, K, and V projections, scaled dot-product attention, masking, softmax, and the weighted sum of values with tensor shapes.

2
Explain the architecture

For multi-head attention, show how heads split/project dimensions and are concatenated before the output projection.

3
Compare trade-offs

Connect architecture to positional information, residuals, normalization, feed-forward blocks, and the training objective.

4
Close with serving + evaluation

Discuss O(n²) attention cost, KV caching for autoregressive decoding, numerical stability, and serving latency.

A useful interview mental model

This is the shape of a strong answer—not a script to memorize.

01Definition
02Mechanism
03Trade-offs
04Failure modes
05When to use

Senior-level signal

  • Reason about memory bandwidth and KV-cache growth, not only FLOPs, when discussing real inference performance.
  • Compare dense attention with sparse/windowed/linear alternatives only after stating the task and context-length trade-off.

What the interviewer is really testing

Understanding beyond prompting: architecture, retrieval, evaluation, inference, safety, latency, cost, and failure recovery.

Likely follow-up questions

What assumption makes this approach work?
When would you choose the strongest alternative instead?
What production or data failure mode changes your answer?

Common weak-answer patterns

  • Reciting a definition without mechanism or assumptions.
  • Claiming one technique is always better without a data regime.
  • Stopping before failure modes, validation, or deployment implications.