GenAI interview guides / LLM Evaluation Interview Guide
LLM Evaluation Interview Guide
Build evaluation systems for generative AI that combine task metrics, model graders, humans, and online signals.
Failure taxonomy
Define product-specific failures before choosing a score. Averages can hide hallucination, safety, latency, or task-completion regressions.
Offline evaluation
Use representative datasets, adversarial slices, deterministic checks where possible, and rubric graders calibrated against humans.
Online evaluation
Track user outcomes, corrections, abandonment, latency, cost, and sampled review. Separate model changes from traffic-mix changes.
Release discipline
Version prompts, models, tools, data, graders, and policies; keep important regressions as permanent tests.