The 60-second answer
Write the forward computation as a graph of local operations and compute the scalar loss first. Starting at the loss, propagate derivatives backward using the chain rule and reuse cached forward activations.
Build the answer in this order
1
Start with intuition
Write the forward computation as a graph of local operations and compute the scalar loss first.
2
Write the mathematical object
Starting at the loss, propagate derivatives backward using the chain rule and reuse cached forward activations.
3
State assumptions
For a linear layer, connect local derivatives to outer products/matrix multiplications; for activations, multiply by the activation derivative elementwise.
4
Connect back to ML behavior
Check gradient shapes and use a finite-difference gradient check on a tiny example to validate the derivation.
A useful interview mental model
This is the shape of a strong answer—not a script to memorize.
01Definition
02Mechanism
03Trade-offs
04Failure modes
05When to use
Senior-level signal
- Senior answers discuss vector-Jacobian products rather than materializing full Jacobians.
- Mention vanishing/exploding gradients and how residual connections, normalization, initialization, and clipping address them.
What the interviewer is really testing
Whether you can connect the math to optimization, uncertainty, model behavior, and a practical engineering decision.
Likely follow-up questions
What assumption makes this approach work?
When would you choose the strongest alternative instead?
What production or data failure mode changes your answer?
Common weak-answer patterns
- Reciting a definition without mechanism or assumptions.
- Claiming one technique is always better without a data regime.
- Stopping before failure modes, validation, or deployment implications.