Husain and Shankar note that coding agents like Claude Code appear to skip formal evals ('we just vibe') without actually skipping the underlying rigor: they inherit it from two sources — (1) the foundation model's own extensive benchmarking, and (2) an expert developer user base that constantly dogfoods the product and surfaces failures immediately. This substitutes for a formal Error Analysis (Manual Trace Review) / LLM-judge loop.
They're explicit that this shortcut doesn't generalize: it only works in domains with a natural population of expert users dogfooding the product daily. Domains lacking that — their example is medicine — can't rely on incidental expert feedback and still need the full error-analysis-driven eval process.
Source: Why AI Evals Are the Hottest New Skill for Product Builders — Hamel Husain & Shreya Shankar.
Hamel Husain and Shreya Shankar's clearest example: coding-agent companies can lean on dogfooding harder than most AI products because 'coding agents are fundamentally very different than other AI products — the developer is the domain expert.' Engineers who build the tool also use it all day, so their own judgment substitutes for a formal domain-expert reviewer. The caveat the speakers give explicitly: this doesn't generalize to domains where builders and expert users don't naturally overlap — medicine is their counter-example — so dogfooding-as-substitute is a special case, not a general alternative to Error Analysis (Manual Trace Review).
Linear's team treats real internal dogfooding feedback as their primary source of eval cases, rather than designing an eval suite speculatively before shipping. The reasoning follows from Ship Early, Let Usage Define Scope: since real usage is what reveals the agent's actual scope and failure modes, it's also the most reliable source of what's worth evaluating — evals emerge from what dogfooders hit, rather than from anticipating problems in advance.