The 60-second answer
Word tokenization is intuitive but has vocabulary/OOV problems; character tokenization has tiny vocabulary but much longer sequences; subword methods balance both. Explain how BPE/WordPiece-style tokenizers trade vocabulary size against sequence length and how rare words, morphology, code, and multilingual text affect fragmentation.
Build the answer in this order
Word tokenization is intuitive but has vocabulary/OOV problems; character tokenization has tiny vocabulary but much longer sequences; subword methods balance both.
Explain how BPE/WordPiece-style tokenizers trade vocabulary size against sequence length and how rare words, morphology, code, and multilingual text affect fragmentation.
Connect tokenization to compute: longer sequences increase attention cost and latency, while larger vocabularies increase embedding/output-layer memory.
Evaluate with downstream task quality plus fertility/fragmentation, unknown-token behavior, language slices, and inference cost.
A useful interview mental model
This is the shape of a strong answer—not a script to memorize.
Senior-level signal
- Senior answers discuss tokenizer/model coupling: changing a tokenizer after pretraining is not a free preprocessing change.
- Include domain adaptation issues such as medical terms, code, IDs, and multilingual scripts.
What the interviewer is really testing
Likely follow-up questions
Common weak-answer patterns
- Reciting a definition without mechanism or assumptions.
- Claiming one technique is always better without a data regime.
- Stopping before failure modes, validation, or deployment implications.