The 60-second answer
Gradient descent updates parameters in the direction opposite the loss gradient; SGD estimates that gradient from a mini-batch. Adam adds adaptive first- and second-moment estimates, often giving fast initial optimization, while SGD can generalize well with careful schedules.
Build the answer in this order
Gradient descent updates parameters in the direction opposite the loss gradient; SGD estimates that gradient from a mini-batch.
Adam adds adaptive first- and second-moment estimates, often giving fast initial optimization, while SGD can generalize well with careful schedules.
Learning rate is usually the most important optimizer hyperparameter; warmup, decay, or plateau schedules can stabilize training.
Diagnose divergence, plateaus, exploding gradients, and noisy updates using loss/gradient statistics.
A useful interview mental model
This is the shape of a strong answer—not a script to memorize.
Senior-level signal
- Discuss optimizer-state memory and distributed-training communication, not only convergence speed.
- For large models, connect schedule/optimizer choices to batch size and effective token budget.
What the interviewer is really testing
Likely follow-up questions
Common weak-answer patterns
- Reciting a definition without mechanism or assumptions.
- Claiming one technique is always better without a data regime.
- Stopping before failure modes, validation, or deployment implications.