Lore

Code-Based (Reference-Based) Evaluation

A deterministic, assertion-style eval with a definitive right answer — no LLM judge required. Example: checking that an extracted date field exactly equals the expected date.

Code-based evals are also called reference-based evals, because they check output against a fixed, known-correct reference. This is contrasted with reference-free evals — LLM-as-judge evals — which score subjective or hard-to-rule-based failures (e.g. human-handoff quality) without a single correct answer to compare against.

The distinction matters operationally: reference-based/code evals are "set and forget" once written, while reference-free/LLM-judge evals need ongoing re-validation against human labels as the judge, model, or product drifts.

Apply: for any failure category surfaced during Axial Coding that has a clear correct answer, write a code assertion instead of building and maintaining an LLM judge — it's cheaper and doesn't need continual re-calibration. See Eval Investment Triage for the full decision between code eval, LLM judge, and no formal eval.

When to Use Code Instead of a Judge

Failure modes that are binary/deterministic and cheap to check — valid JSON, correct format, a required field present — should be caught with a plain code assertion rather than an LLM judge. This is the counterpart to LLM-as-Judge Validation, which is reserved for failure modes that are subjective or otherwise resist a hard-coded check (e.g., whether a human handoff was warranted).

Pairing With Subjective Checks (Linear)

Linear's agent evals mix deterministic, code-based checks — e.g., a ticket's status field is always set correctly — with subjective quality checks scored by an LLM-as-Judge Validation, e.g., whether a generated description is well-structured. The deterministic half catches hard facts that must always hold; the subjective half is reserved for dimensions where quality is real but not reducible to a fixed reference answer. See consistency-scoped-eval-principle for the criterion Linear uses to decide which half a given behavior belongs to.