CrackML by @ml.with.umang
Interview questions / ML Coding & PyTorch
ML Coding & PyTorch interview question

Implement a Transformer Block

Implement the core forward pass of a Transformer block with attention, residual connections, normalization, and an MLP.

hardml codingEvidence 34/1001 source reportTikTok

The 60-second answer

Define tensor shapes first: batch, sequence, hidden size, number of heads, and per-head dimension; assert hidden_size % heads == 0. Implement normalized attention → residual and normalized MLP → residual (or clearly state post-norm if using that variant), including dropout/masks where required.

Build the answer in this order

1
State tensor contract

Define tensor shapes first: batch, sequence, hidden size, number of heads, and per-head dimension; assert hidden_size % heads == 0.

2
Implement the mechanism

Implement normalized attention → residual and normalized MLP → residual (or clearly state post-norm if using that variant), including dropout/masks where required.

3
Check numerics + gradients

Reshape/transposes must preserve batch/head/sequence semantics and return the original [B,T,D] shape.

4
Test shapes and edge cases

Test with tiny deterministic tensors, causal/padding masks, train/eval modes, and a backward pass that produces finite gradients.

A useful interview mental model

This is the shape of a strong answer—not a script to memorize.

01Shapes
02Forward pass
03Loss / grads
04Numerics
05Tests

Senior-level signal

  • Senior answers discuss pre-norm stability, fused attention kernels, and avoiding unnecessary contiguous copies/transposes.
  • Call out KV caching as a serving concern, separate from the training forward pass.

What the interviewer is really testing

Tensor fluency, shape reasoning, numerics, gradients, batching, device awareness, and the ability to debug—not API memorization.

Likely follow-up questions

What are the tensor shapes at each step?
Where could numerical instability or silent broadcasting appear?
How would you verify gradients and batched behavior?

Common weak-answer patterns

  • Ignoring shape, dtype, device, masking, or broadcasting assumptions.
  • Using a framework call without explaining the underlying operation.
  • Skipping gradient, numerical-stability, and batching checks.