The 60-second answer
Sigmoid maps values to (0,1) and is natural for binary probabilities at an output layer, but saturates and can produce small gradients in deep hidden layers. ReLU is cheap and non-saturating on the positive side, making it a common hidden activation, though units can become permanently inactive.
Build the answer in this order
Sigmoid maps values to (0,1) and is natural for binary probabilities at an output layer, but saturates and can produce small gradients in deep hidden layers.
ReLU is cheap and non-saturating on the positive side, making it a common hidden activation, though units can become permanently inactive.
Mention alternatives such as GELU/SiLU/leaky ReLU when smoothness or dead-unit behavior matters.
Choose output activation from the probabilistic objective and hidden activation from optimization/architecture evidence.
A useful interview mental model
This is the shape of a strong answer—not a script to memorize.
Senior-level signal
- Senior answers separate activation choice from loss semantics: sigmoid plus BCE is different from using sigmoid everywhere.
- Discuss mixed precision and fused kernels only when system constraints make them relevant.
What the interviewer is really testing
Likely follow-up questions
Common weak-answer patterns
- Reciting a definition without mechanism or assumptions.
- Claiming one technique is always better without a data regime.
- Stopping before failure modes, validation, or deployment implications.