The 60-second answer
BERT is an encoder-only Transformer: every token can attend bidirectionally to the full unmasked input context. Its original pretraining used masked-language modeling and next-sentence prediction; downstream tasks typically fine-tune the encoder or add lightweight task heads.
Build the answer in this order
BERT is an encoder-only Transformer: every token can attend bidirectionally to the full unmasked input context.
Its original pretraining used masked-language modeling and next-sentence prediction; downstream tasks typically fine-tune the encoder or add lightweight task heads.
Contrast it with decoder-only models, which use causal masking and are naturally suited to autoregressive generation.
Mention practical tradeoffs: BERT-style encoders remain strong for classification, token labeling, and dense representations where generation is unnecessary.
A useful interview mental model
This is the shape of a strong answer—not a script to memorize.
Senior-level signal
- Senior answers distinguish pretraining objective from architecture and avoid implying NSP is essential to all BERT-family models.
- Discuss when a smaller encoder is materially cheaper and more reliable than prompting a large generative model.
What the interviewer is really testing
Likely follow-up questions
Common weak-answer patterns
- Reciting a definition without mechanism or assumptions.
- Claiming one technique is always better without a data regime.
- Stopping before failure modes, validation, or deployment implications.