The 60-second answer
Smaller batches produce noisier gradient estimates and more optimizer steps per example; larger batches improve hardware utilization but need more memory and can reduce update frequency. Learning rate and schedule often need to change with effective batch size; gradient accumulation changes memory footprint without changing the statistical batch.
Build the answer in this order
Smaller batches produce noisier gradient estimates and more optimizer steps per example; larger batches improve hardware utilization but need more memory and can reduce update frequency.
Learning rate and schedule often need to change with effective batch size; gradient accumulation changes memory footprint without changing the statistical batch.
In distributed training, larger batches amortize communication but can increase synchronization and convergence tradeoffs.
Compare time-to-quality at fixed compute, not just examples/second or loss per step.
A useful interview mental model
This is the shape of a strong answer—not a script to memorize.
Senior-level signal
- Senior answers discuss the distinction between micro-batch, local batch, and global effective batch.
- Include gradient accumulation, mixed precision, and optimizer-state memory when reasoning about feasible scale.
What the interviewer is really testing
Likely follow-up questions
Common weak-answer patterns
- Reciting a definition without mechanism or assumptions.
- Claiming one technique is always better without a data regime.
- Stopping before failure modes, validation, or deployment implications.