The 60-second answer
Use log-sum-exp: subtract the per-row max, compute logsumexp, then select the target logit to obtain negative log likelihood. Avoid explicitly forming probabilities when they are unnecessary; this improves numerical stability.
Build the answer in this order
1
State tensor contract
Use log-sum-exp: subtract the per-row max, compute logsumexp, then select the target logit to obtain negative log likelihood.
2
Implement the mechanism
Avoid explicitly forming probabilities when they are unnecessary; this improves numerical stability.
3
Check numerics + gradients
Support a clear reduction contract (none/mean/sum) and validate label bounds and tensor shapes.
4
Test shapes and edge cases
Compare forward values and gradients against the framework implementation on random and extreme logits.
A useful interview mental model
This is the shape of a strong answer—not a script to memorize.
01Shapes
02Forward pass
03Loss / grads
04Numerics
05Tests
Senior-level signal
- Senior answers discuss ignore-index, class weights, label smoothing, and distributed reduction semantics.
- Test mixed-precision behavior and keep critical reductions in a stable dtype when needed.
What the interviewer is really testing
Tensor fluency, shape reasoning, numerics, gradients, batching, device awareness, and the ability to debug—not API memorization.
Likely follow-up questions
What are the tensor shapes at each step?
Where could numerical instability or silent broadcasting appear?
How would you verify gradients and batched behavior?
Common weak-answer patterns
- Ignoring shape, dtype, device, masking, or broadcasting assumptions.
- Using a framework call without explaining the underlying operation.
- Skipping gradient, numerical-stability, and batching checks.