Lore

Evals as the Modern PRD ("What Not How")

Overview

Ankur Goyal (CEO, Braintrust) argues that a well-built eval is the contemporary equivalent of a product requirements document (PRD): a traditional PRD states success in prose and examples for a human engineer to interpret; an eval makes that same "what does success look like" statement quantifiable and machine-checkable, so a model can own the "how" autonomously. This extends a shift machine learning already made — regression and transformer-style training already moved programming from specifying computation steps to specifying a desired outcome — and Goyal generalizes that "what not how" framing to any AI-assisted work.

Beyond AI-output quality: hard evals for general engineering

The same technique isn't limited to judging AI product outputs — Goyal argues the highest-leverage move once models write code well is building hard, rigorous evals for any technical problem, including infrastructure work that has nothing to do with AI quality. Braintrust's own database query-performance effort illustrates this: instead of manually optimizing, the team mines patterns from real slow queries and reproduces them as concrete test cases, turning a fuzzy "why is this slow" complaint into a benchmark an agent can iterate against exhaustively (see Agent-Driven Exhaustive Experimentation (Practical vs. Theoretical Quality) for what the agent then does with that benchmark). Once a problem has a hard, reproducible success criterion, an agent can work on it creatively and autonomously in the background instead of being told the solution steps.

Related

Criteria Drift and the Living PRD

Husain and Shankar reframe a well-built LLM-judge prompt as a living, continuously running product requirements document: it is grounded in real data (via Error Analysis (Manual Trace Review)) and gets updated as reviewers encounter failure modes they couldn't have anticipated upfront — a phenomenon they call criteria-drift. They still recommend starting from a traditional PRD, treating the judge prompt as the artifact that keeps that PRD current rather than as a replacement for writing one initially.

PRD Is Still Written First

Hamel Husain and Shreya Shankar don't propose skipping a traditional PRD — write it before building the product as usual. The judge prompt then functions as a living, data-driven refinement of that PRD: it encodes conditional "if this, then that" behavioral expectations that run automatically and continuously against real traffic, rather than sitting as a static document. As Error Analysis (Manual Trace Review) surfaces expectations nobody anticipated, the judge prompt gets updated — this is the same phenomenon named criteria-drift. Quote: "This is like the purest sense of what a product requirements document should be is this eval judge that's telling you exactly what it should be and it's automatic and running constantly."