Eval outputs should be collapsed to binary pass/fail wherever possible, rather than continuous or Likert-style scales (e.g., 1–5). Aggregate scores like an average of 3.2 vs 3.7 are not actionable — no one can say what the difference means or what to do about it.
Narrow, component-level evals — e.g., retrieval quality — are an acknowledged exception where a non-binary score can still be useful.
Binary scoring is the recommended output format for validated LLM judges built from failures surfaced in Error Analysis (Manual Trace Review).
LLM judges should output true/false, not a 1–5 or continuous score. Quote: "It's very important for an LLM judge that you output a binary score." Continuous and Likert-style scores are harder for LLMs to apply consistently and are less actionable — a "3 out of 5" doesn't tell you whether to ship. Frame the judge prompt around a single yes/no threshold question, e.g. "is this good enough to ship?", rather than a graded rubric. Reserve numeric/graded scores for narrow, component-level evals (e.g. retrieval ranking) where a scale is meaningful.
Pair the binary verdict with a required explanation field: prompt the judge to always justify its true/false call in free text alongside the score. This makes it possible to debug why the judge flagged or missed a specific failure, rather than treating it as an opaque classifier. See LLM-as-Judge Validation for how this binary output gets checked against human labels via TPR/TNR.
Hamel Husain and Shreya Shankar argue LLM judges should never be asked for a 1–5 or 1–7 rating: scales let an evaluator avoid making a clear call, and the resulting averages are not actually interpretable as a quality signal. Forcing a true/false verdict is what makes a judge usable both as a CI check and inside LLM-as-Judge Validation.