The 60-second answer
Contrast encoder-only bidirectional attention with decoder-only causal attention and state the corresponding pretraining objectives. Explain why BERT naturally produces contextual representations while GPT naturally supports autoregressive generation.
Build the answer in this order
Contrast encoder-only bidirectional attention with decoder-only causal attention and state the corresponding pretraining objectives.
Explain why BERT naturally produces contextual representations while GPT naturally supports autoregressive generation.
Compare fine-tuning/inference interfaces, masking, context direction, and common downstream use cases.
Discuss evaluation and serving trade-offs rather than reducing the comparison to model size.
A useful interview mental model
This is the shape of a strong answer—not a script to memorize.
Senior-level signal
- Tie architecture choice to latency, context length, output type, and whether the product needs generation or representation learning.
- Mention that modern variants blur simple families, so objective and attention mask are more fundamental than model branding.
What the interviewer is really testing
Likely follow-up questions
Common weak-answer patterns
- Reciting a definition without mechanism or assumptions.
- Claiming one technique is always better without a data regime.
- Stopping before failure modes, validation, or deployment implications.