CrackML by @ml.with.umang
Interview questions / GenAI & LLM
GenAI & LLM interview question

Explain embeddings, tokenization, attention, and BERT

Walk through how text becomes model inputs: tokenization, embeddings, self-attention, and BERT-style contextual representations. Explain shapes, training objectives, and common tradeoffs.

mediumconceptEvidence 41/1001 source reportLinkedIn

The 60-second answer

Tokenization converts text into token IDs; embedding tables map IDs to dense vectors and positional information encodes order. Self-attention projects Q/K/V, computes softmax(QKᵀ/√d), applies masks, and mixes V; multi-head attention repeats this in learned subspaces.

Build the answer in this order

1
Define the mechanism

Tokenization converts text into token IDs; embedding tables map IDs to dense vectors and positional information encodes order.

2
Explain the architecture

Self-attention projects Q/K/V, computes softmax(QKᵀ/√d), applies masks, and mixes V; multi-head attention repeats this in learned subspaces.

3
Compare trade-offs

BERT is an encoder-only Transformer that produces bidirectional contextual representations and was originally pretrained with masked-language modeling.

4
Close with serving + evaluation

Connect the components to concrete use cases such as classification, retrieval, or token labeling instead of reciting definitions.

A useful interview mental model

This is the shape of a strong answer—not a script to memorize.

01Definition
02Mechanism
03Trade-offs
04Failure modes
05When to use

Senior-level signal

  • Discuss subword fragmentation, O(n²) attention cost, and padding/attention-mask semantics.
  • Distinguish token embeddings from retrieval embeddings and encoder-only models from decoder-only generation.

What the interviewer is really testing

Understanding beyond prompting: architecture, retrieval, evaluation, inference, safety, latency, cost, and failure recovery.

Likely follow-up questions

What assumption makes this approach work?
When would you choose the strongest alternative instead?
What production or data failure mode changes your answer?

Common weak-answer patterns

  • Reciting a definition without mechanism or assumptions.
  • Claiming one technique is always better without a data regime.
  • Stopping before failure modes, validation, or deployment implications.