The 60-second answer
State inputs, shapes, invariants, and numerical assumptions before coding. Implement the simplest correct version first, then optimize the measured bottleneck.
Build the answer in this order
1
State tensor contract
State inputs, shapes, invariants, and numerical assumptions before coding.
2
Implement the mechanism
Implement the simplest correct version first, then optimize the measured bottleneck.
3
Check numerics + gradients
Test edge cases against a trusted small reference implementation.
4
Test shapes and edge cases
Explain time, memory, and numerical-stability trade-offs.
A useful interview mental model
This is the shape of a strong answer—not a script to memorize.
01Shapes
02Forward pass
03Loss / grads
04Numerics
05Tests
Senior-level signal
- For GPU code, distinguish compute-bound from memory-bandwidth-bound operations.
- Use deterministic seeds and gradient or numerical checks where appropriate.
What the interviewer is really testing
Tensor fluency, shape reasoning, numerics, gradients, batching, device awareness, and the ability to debug—not API memorization.
Likely follow-up questions
What are the tensor shapes at each step?
Where could numerical instability or silent broadcasting appear?
How would you verify gradients and batched behavior?
Common weak-answer patterns
- Ignoring shape, dtype, device, masking, or broadcasting assumptions.
- Using a framework call without explaining the underlying operation.
- Skipping gradient, numerical-stability, and batching checks.