AI evaluations
Hamel Husain argues, using a real Nurture Boss case study, that manually reading a modest sample of raw AI conversation traces and simply annotating what went wrong is the single highest-value and most commonly skipped step in AI evaluation — and that only after doing this can teams sensibly decide which problems deserve a cheap code-based check, which deserve a validated LLM-as-judge, and which need no formal eval at all.
The video's problem framing: eval programs commonly rely on vague aggregate scores and 'agreement' percentages that no one can act on or trust, and once trust breaks, the whole evals effort loses credibility with stakeholders.
Its central claim: manually reading a modest sample of real traces and simply noting what's wrong is the highest-value, most commonly skipped step in building evals — more informative than throwing generic automated scores at the data.
It supports this with the Nurture Boss case study: a client initially saw manual trace-reading as a waste of an 'expensive consulting engagement,' but the exercise surfaced specific, real failures that off-the-shelf hallucination/toxicity/coherence scores would have missed entirely.
It argues that not every discovered problem warrants a formal eval: cheap, obviously-fixable issues (e.g., feeding today's date to the model) should just be fixed, while only recurring, harder-to-resolve issues (e.g., human handoff) justify the cost of an LLM-judge eval.
It argues LLM-as-judge evals are not inherently trustworthy and must themselves be validated against human labels before anyone relies on their output — otherwise a big enough gap between judge output and reality will eventually destroy stakeholder trust.
It argues that 'agreement' is a deceptive metric because real-world failure rates are typically low, so a lazy always-predict-no-failure judge can look highly accurate while being useless; disaggregated true-positive/true-negative rates and a confusion matrix are presented as the honest alternative.
It argues for collapsing eval outputs to binary pass/fail wherever possible, on the grounds that continuous/Likert scales create false precision, are hard for LLMs to produce reliably, and let teams avoid making a real ship/no-ship decision — with narrow component-level evals (e.g., retrieval) as an acknowledged exception.
It frames the manual annotation/error-analysis step as a continuous, ongoing discipline rather than a one-off, to be repeated on a cadence and supported with lightweight custom tooling, because that is where most of the value lives.
It criticizes the generic, 'plug-and-play' vendor approach to evals (helpfulness/toxicity dashboards) as usually mismatched to a specific application's actual problems, while conceding generic scores can still be repurposed as a sampling device to direct further manual analysis.
It scales the prescription by context: pre-launch or low-usage products can rely on the builder's own hands-on use as informal error analysis, while formal evals become necessary only once usage outgrows what one person can personally track, with gradual rollouts used to gather real data safely.
It closes by tying 'look at your data' — common but vague advice — concretely back to the counting/annotation practice described throughout, positioning it as inseparable from, rather than merely preliminary to, doing evals at all.
Manual trace review / error analysis — Manually reading a sample of roughly 100 raw conversation traces and writing a short note on what's wrong with each, without trying to root-cause or fix it yet. Apply: Read through around 100 traces per product/feature, jot a one-line note of the failure per trace, and expect the pace to speed up markedly by around the 10th trace.
Open coding — The raw, unstructured notes taken while reading each trace during error analysis, a term borrowed from qualitative social-science research and long used in ML. Apply: Capture whatever you notice going wrong on each trace as a free-text 'open code' before trying to organize it into categories.
Axial coding — Grouping and classifying the open codes into a smaller set of named categories, such as tour-scheduling issues, human-handoff issues, or formatting errors. Apply: Feed the collected open codes to an LLM and ask it to propose five to six axial-code categories, then label each trace's open code with the category it belongs to.
Dimension-based synthetic data generation — Generating synthetic user queries by first defining orthogonal dimensions (e.g., customer type × product tier) and crossing them into combinations, rather than asking an LLM for generic plausible questions. Apply: Brainstorm two or more dimensions relevant to your product's personas or segments, enumerate the combinations, and prompt an LLM to generate queries for each combination separately to get diverse coverage.
LLM-as-judge — Using an LLM prompted with explicit failure criteria to score a trace, typically for subjective or hard-to-rule-based problems such as human-handoff quality. Apply: Write a prompt defining exactly what counts as a failure (e.g., a list of concrete handoff-failure conditions) and ask the judge to output a binary true/false plus an explanation.
Code-based (reference-based) eval — A deterministic assertion-style check with a definitive right answer, requiring no LLM judge, such as checking that an output date equals the expected date. Apply: For issues with a clear correct answer, write a simple code assertion test case instead of building an LLM judge, since it's cheaper and needs no ongoing human-label maintenance.
Reference-based vs. reference-free eval terminology — A naming convention distinguishing code-based evals with a fixed right answer (reference-based) from LLM-judge evals without one (reference-free). Apply: Use this vocabulary with your team to flag which evals need continual re-calibration (reference-free/LLM-judge) versus which are 'set and forget' (reference-based/code).
Meta-evaluation ('measuring the judge') — Validating an LLM judge's own accuracy against human-labeled ground truth before trusting its output, since a judge is only as good as its match to human labels. Apply: Compare the judge's binary verdicts on the same traces against human labels (obtained for free from axial coding) before relying on the judge in production.
Binary scoring over Likert/continuous scales — The recommendation that LLM judges output true/false rather than 1-5 or continuous scores, since LLMs handle continuous scores poorly and such scores aren't actionable. Apply: Reframe judge prompts and rubrics around a single yes/no threshold question ('is this good enough to ship?') instead of a graded scale, reserving scores for narrow component-level evals like retrieval.
Structured judge output (explanation + binary score) — Having the LLM judge return two fields: a free-text explanation and a binary verdict. Apply: Prompt the judge to always justify its true/false call in an explanation field so you can debug why it flagged or missed a failure.
Confusion matrix — A 2x2 table comparing human labels against LLM-judge labels on true/false axes to see where the judge agrees or disagrees. Apply: Tabulate your labeled traces into the four cells (true/true, true/false, false/true, false/false) to visually inspect where the judge is failing, rather than trusting a single aggregate number.
True Positive Rate / True Negative Rate (sensitivity/specificity) — TPR measures how well the judge catches real failures when they occur; TNR measures how well it correctly clears non-failures; together they replace raw 'agreement' as a diagnostic. Apply: Compute TPR and TNR separately from your confusion matrix and treat both as the pass/fail criteria for whether a judge is good enough, rather than one blended agreement percentage.
'Agreement' as a red-flag metric — Raw percent-agreement between judge and human labels, which is misleading when failures are rare because always predicting 'no failure' yields high agreement. Apply: Whenever a team reports 'agreement' as an eval quality metric, ask for the underlying base failure rate and push for TPR/TNR instead.
Few-shot examples in judge prompts with train/held-out split — Adding example-labeled traces to a judge prompt to improve accuracy, which risks overfitting if the same traces are also used to measure judge accuracy. Apply: Split your labeled traces into a set used as few-shot examples in the prompt and a separate held-out set used only to measure the judge's TPR/TNR.
CI and production monitoring integration for judges — Deploying validated judges both in continuous integration to test every code/prompt change, and on a sample of live production traffic to catch and track ongoing failures. Apply: Once a judge's TPR/TNR are acceptable, wire it into CI test suites and also run it on a sampled percentage of production conversations to build an ongoing monitoring dashboard.
Custom annotation tooling — A purpose-built, lightweight internal tool (trace viewer with channel selectors, hidden system prompt, note field, built-in axial coding and counting) to speed up manual trace review. Apply: Build a minimal bespoke annotation UI (the Nurture Boss example took under four hours) once manual review proves valuable, so labeling becomes fast enough to sustain regularly.
Theoretical saturation — A qualitative-research stopping rule: keep reviewing traces until you experience diminishing returns and stop learning anything new. Apply: Use around 100 traces as a starting anchor for manual review, but keep going past that only until you notice you're no longer discovering new failure patterns.
Generic metrics as a sampling/exploration tool — Repurposing generic off-the-shelf scores (e.g., a hallucination score) not as reported metrics but as a way to sort traces and direct manual error analysis toward promising cases. Apply: Score all traces with a generic metric, sort by that score, and manually error-analyze the highest or lowest scoring traces to see if a real pattern emerges — never report the generic score itself as a result.
Spreadsheet-based AI formulas for categorization/judging — Using a spreadsheet's built-in AI formula feature (e.g., Google Sheets) to run categorization or LLM-judge prompts directly on rows of data without writing code. Apply: Put each trace or note in a spreadsheet row and use an AI formula column to categorize it or apply a judge prompt, making the whole pipeline accessible to non-engineers.
The claimed 90%-agreement-while-useless illustration: with a real failure rate of roughly 10-15%, a judge that never flags failures can still show about 90% 'agreement,' a concrete trap the video says many teams fall into.
Human-labeled ground truth for calibrating a judge doesn't require separate labeling effort — it falls out 'for free' from the axial-coding categories already produced during manual error analysis.
Adding few-shot examples to a judge prompt without splitting data into train/held-out sets lets the judge 'cheat' and score 100%, since it's just recognizing traces it already memorized.
The '100 traces' figure is presented not as a rigorous threshold but as a motivating round number to get people started; the actual stopping criterion given is 'theoretical saturation' — stop once you're no longer learning anything new.
Generic evaluators like a hallucination score are given a legitimate secondary use as a sorting device to direct manual review toward the most interesting traces, even though the video says they should never be reported as metrics themselves.
Once a team builds even a bare-bones custom annotation tool (the Nurture Boss example was built in under four hours), the pace of manual labeling becomes fast enough that people reportedly get 'addicted' to doing it, reversing the usual reluctance toward the tedious parts of eval work.
Synthetic test data generated from a single, generic 'come up with plausible questions' prompt is described as producing homogeneous, low-diversity output; crossing explicit dimensions (customer type × product tier, etc.) is presented as the fix.
«this looking at data accounting. You can get insane value by just doing that. And that's the one part that everyone skips.»
— 00:00
«When you see an average score of 3.2 versus 3.7, no one really knows what the hell that means.»
— 00:13
«If people don't trust your evals, they won't even trust you. You're done.»
— 00:33
«if you tried to come at nurture boss and put a hallucination score, toxicity score, coherence score, whatever you want to call it, it would not have given us anywhere close to this insight that we had right now.»
— 09:03
«You don't want to just ask an LLM, hey, come up with plausible questions for a prospective tenant that might be looking to rent an apartment. The right way to do it is to come up with some categories... some dimensions.»
— 16:08
«There's two kinds of evals. One is LM as a judge... Another one is codebased eval...»
— 20:22
«It's very important for an LLM judge that you output a binary score.»
— 24:41
«You can agree with it 90% of the time by just saying it never fails.»
— 29:43
«If I put all of these traces in my prompt, it would get 100%.»
— 34:48
«The term is called theoretical saturation and that just means you keep doing the activity until... it's like a diminishing returns... until you aren't learning anything new.»
— 42:52
«Then nothing beats real data.»
— 47:11
«It's not like the super sexy part of it, but yeah, it's the most important part of it.»
— 49:49
Reception
Viewers overwhelmingly praised the video as practical, insightful, and among the best content on evals, with no negative comments.
The video functions as a hands-on tutorial that translates the generic advice 'look at your data' into a specific, reproducible methodology (open/axial coding, judge validation via TPR/TNR, binary scoring) anchored in a single real client engagement rather than abstract principles.

52:30