The 60-second answer
RLHF typically learns a reward/preference signal and then optimizes the policy against it, often with a KL constraint to a reference model. DPO directly optimizes a preference objective from chosen/rejected pairs without an explicit reward-model-plus-RL stage.
Build the answer in this order
RLHF typically learns a reward/preference signal and then optimizes the policy against it, often with a KL constraint to a reference model.
DPO directly optimizes a preference objective from chosen/rejected pairs without an explicit reward-model-plus-RL stage.
DPO is operationally simpler, while RLHF offers more flexibility for complex reward shaping and online interaction.
Both depend strongly on preference-data quality, coverage, and distribution shift.
A useful interview mental model
This is the shape of a strong answer—not a script to memorize.
Senior-level signal
- Discuss reward hacking, annotator disagreement, and KL/reference-model choices rather than treating preference optimization as a clean supervised problem.
- Evaluate capability and safety regressions on slices outside the preference dataset.
What the interviewer is really testing
Likely follow-up questions
Common weak-answer patterns
- Reciting a definition without mechanism or assumptions.
- Claiming one technique is always better without a data regime.
- Stopping before failure modes, validation, or deployment implications.