Multimodal LLM Systems Interview Guide
Design production systems that combine text, image, audio, or video inputs with modality-specific preprocessing, fusion, evaluation, and serving.
Input pipeline
Treat modality preprocessing as part of the model contract: image resizing/cropping, video frame sampling, audio segmentation, text tokenization, ordering, and metadata all affect quality and latency.
Fusion architecture
Explain whether modalities are encoded separately and projected into a shared representation, fused through cross-attention, or represented as model tokens. Tie the choice to task quality, context length, and serving cost.
Evaluation
Build modality-specific slices: OCR-heavy images, low-light scenes, long videos, noisy audio, contradictory text/image evidence, missing modalities, and safety-sensitive content. Evaluate both perception and final task success.
Serving and failure handling
Account for large media payloads, preprocessing queues, accelerator memory, caching, partial modality failure, privacy controls, and graceful degradation when one modality is unavailable.