Lore

Agent-Driven Exhaustive Experimentation (Practical vs. Theoretical Quality)

Overview

Ankur Goyal (CEO, Braintrust) distinguishes a human engineer's theoretical quality ceiling — the benchmarks and algorithms they could in principle run — from their practical quality, which decays with attention and tedium over hard, exhaustive work. His claim: integrating agents raises practical quality specifically because an agent sustains rigor longer and more consistently than a person does, turning exhaustive experimentation that was previously too expensive to justify (e.g., a year spent testing six indexing solutions) into tractable, low-risk background work a business will approve, since it runs alongside normal shipping instead of blocking it.

Case study: Braintrust query performance

Braintrust applied this to database query optimization: rather than picking a column-store format and execution engine by intuition, they had an agent exhaustively test the full combinatorial matrix of open-source options against literature-derived techniques. After a week of continuous agent experimentation, this surfaced a Bloom-filter index that ad hoc human "probability" reasoning about complexity had been getting wrong about half the time — evidence that intuition-based triage of which techniques are "probably not worth trying" was an unreliable shortcut agents no longer force engineers to take.

Related

Case Study: Braintrust's Column Store

Braintrust's search/indexing was built on Tantivy, an open-source library whose built-in column store was too weak for their needs. Rather than accepting the default or picking a replacement by reputation, the team ran agent-driven experiments over about a week to test alternatives — including bloom filters, a probabilistic indexing structure with a "bad reputation" in general use. The experiments showed bloom filters worked well for their actual workload, so they adopted them despite the reputation.

A related practice: some of this testing ran directly against real production data (with proper engineering safeguards) rather than only staging or synthetic data, using agents to run the comparisons at a scale and consistency a human wouldn't sustain manually.

"Half the time you're wrong. Now there is no excuse to not do those benchmarks."

This is the concrete version of the broader claim: don't dismiss a data structure or approach because of its reputation — let agent-run experiments test it against the actual workload. See also model-selection-for-pushback for the model used on this kind of hard problem, and protect-maker-time for the working conditions this depth of experimentation assumes.

Case Study: Database Performance and Data Migration

Two supporting cases from Braintrust's own engineering: