Most teams discover they need an eval harness at the worst moment: a model upgrade ships, a handful of customer complaints arrive, and nobody can say whether the new version is better or whether the complaints are about something else. A harness exists to answer that question in minutes, with a number you trust, before the change reaches production.
The goal is not a leaderboard score. It is a repeatable test suite for your specific product, wired into the same pipeline as your unit tests, that catches regressions in quality, cost, and latency at the same time. This guide covers the parts that decide whether the harness survives contact with a real roadmap.
Decide what the harness is for
An eval harness answers three different questions, and mixing them up is the most common design mistake. Does the system do the task at all on a fixed set of cases? Did a change make it worse than the version in production? Is the change worth its cost and latency? The first is capability, the second is regression, the third is an economic decision. Teams that build one giant score usually end up with a number that no one can act on.
Write the questions down before you write code, and give each one its own suite. Capability suites can be large and slow. Regression suites must be small enough to run on every pull request. Economic checks can be a single dashboard. The rest of the design follows from that split.
Build the golden set from production, not from benchmarks
Public benchmarks measure general ability, and they are likely to be contaminated by the time you read about them. LM Evaluation Harness is an excellent way to compare base models, and it will not tell you whether your refund-policy answer is still correct. Your golden set should come from real traffic: sampled conversations, support tickets, documents, and edge cases your team has already argued about.
Start with 150 to 500 cases and stratify them. Cover the common path, the long tail, the cases where the system must refuse, and the cases where it must cite a source. Every case needs three things: an input, an expected behavior described precisely enough to grade, and a note on why the case exists. Cases without a reason get deleted by the next person who touches the file.
Version the set with the code. When a customer reports a failure, add the case before you fix it. That habit turns support into coverage, and it gives you a defensible answer when someone asks whether quality is improving.
One trap to design against: a golden set that the team optimizes against directly stops measuring quality and starts measuring memorization. Keep a held-out slice that nobody tunes on, refresh the set quarterly from new traffic, and track how many cases were added because of a real failure rather than a hunch. A set that never changes is a set that has stopped learning from production.
Grade with code first, models second
Most of what you care about is checkable without a judge. Did the response match the required schema? Did it cite a document that exists? Did the tool call include the right arguments? Did it refuse when the input was out of scope? Did it stay under the token budget? These assertions are cheap, deterministic, and fast. They should catch the majority of regressions, and they will never drift.
Use a model judge for the residue: tone, helpfulness, faithfulness to a retrieved document, adherence to a rubric. A judge is a measurement instrument, not an oracle, and it has to be built like one. Give it a rubric with explicit criteria, force a structured output, and hide the model name from it so it cannot prefer a familiar vendor. Run the same case twice with the answer order swapped when you are comparing two systems side by side.
Treat the judge as an instrument you must calibrate
The evidence on judge quality is mixed in a useful way. The MT-Bench study found that strong judge models can agree with human preferences at rates comparable to agreement between humans in some settings, and it documented the failure modes that matter: position bias, verbosity bias, and a preference for outputs from the same model family. Those biases are systematic, which means they can be measured and partly corrected.
Calibrate before you trust. Label 50 to 100 cases by hand, with two reviewers on the ones that matter most, and compare the judge to those labels. Report agreement, not just accuracy. If the judge agrees with your reviewers less often than your reviewers agree with each other, the judge is not ready to gate a release. Recalibrate whenever you change the judge model or the rubric, because a judge update is a measurement change, not a housekeeping task.
Keep a small human-labeled set permanently. It is the only ground truth you have, and it is the first thing teams skip when they are busy.
Run regression tests in continuous integration
A harness that lives on someone's laptop is a demo. The regression suite should run on every pull request that touches prompts, retrieval, model routing, or tool definitions, and it should fail the build on a real regression. Two things make that practical.
First, control noise. The error-bars work from Anthropic makes the case that eval scores are estimates with sampling variance, so a two-point difference on a few hundred cases is often indistinguishable from chance. Compare paired runs on the same cases, fix the sampling settings, and set thresholds from the observed variance of repeated runs rather than from a round number someone liked. Run each suite three times before you believe a small change.
Second, control cost. A regression suite of 300 cases with a judge call can run in a few minutes and a few dollars, which is cheap enough for every pull request. Capability suites can run nightly. Cache model responses keyed by input, prompt version, and model version so a retry does not pay twice, and keep a seed and a recorded response file so a failure can be reproduced exactly.
Track cost and latency as first-class results
Quality that arrives 40% slower or 3x more expensive is a product decision, not a free win. Record tokens in and out, wall-clock latency at the median and the 95th percentile, tool calls per case, and cost per case for every run. Then chart them next to the quality score. The most useful artifact a harness produces is a table where one candidate is better on quality and worse on cost, because that is the decision the team actually has to make.
Put a budget in the test itself. If a change pushes cost per case above the agreed ceiling or the 95th-percentile latency above the limit, the build fails with the same authority as a broken assertion. This is the difference between a harness and a report.
The harness checklist
| Layer | What it catches | Cadence | Gates the merge |
|---|---|---|---|
| Deterministic assertions | Schema breaks, missing citations, bad tool arguments, missing refusals | Every commit | Yes |
| Golden-set regression | Quality drift on real traffic, including known past failures | Every pull request | Yes |
| Judge-scored rubric | Tone, faithfulness, helpfulness on hard cases | Every pull request | Yes, above a variance-based threshold |
| Adversarial suite | Prompt injection, out-of-scope requests, unsafe outputs | Nightly and before releases | Yes |
| Cost and latency budget | Token inflation, extra tool calls, slow paths | Every pull request | Yes |
| Production monitors | Distribution shift that no fixed set can see | Continuous, sampled | No, but alerts |
The order matters. Deterministic checks are cheap and precise, so they run first and narrow the failure before expensive judging begins. Production monitors are deliberately outside the gate, because they catch the case your golden set has not seen yet and feed it back into the set.
What to do next
- Write down the three questions the harness must answer, then create one suite per question.
- Pull 150 real cases this week, label the expected behavior, and give every case a stated reason.
- Add deterministic assertions before you add a judge. Aim for most failures to be caught without a model call.
- Hand-label 50 cases, measure judge agreement, and publish that number next to every score.
- Wire the regression suite into continuous integration with a cost and latency budget, then run it three times to learn its noise before you enforce thresholds.
A harness is not a one-time project. It is the instrument your team uses to make model decisions quickly. Build the smallest version that runs on every change, and let the case count grow from the failures you actually see.
