Evaluating agents beyond benchmarks

If you cannot detect a five-point regression, you are not iterating — you are guessing with extra steps. Statistical power is the first thing to design and the last thing anyone budgets for.

Every team we work with has the same evaluation story. There is a spreadsheet of about thirty examples. Someone runs them by hand after a prompt change and reads the outputs. It worked fine for the first month and now nobody trusts it, because the outputs are long, the differences are subtle, and the person reading them knows which version they are hoping wins.

The fix is not a bigger spreadsheet. It is treating evaluation as a measurement instrument with a known error bar.

Start with the power calculation#

If your evaluation cannot detect the size of improvement you are trying to make, running it is theatre.

For a binary outcome, the standard error on a success rate p over n trials is sqrt(p(1-p)/n). At p = 0.7, that is about 4.6 percentage points at n = 100 and 1.4 points at n = 1000. Detecting a 5-point improvement with any confidence needs several hundred examples, not thirty.

Two techniques buy back a lot of this without more examples:

Paired comparison. Run both variants on the same examples and test the difference per example rather than comparing two independent rates. Example difficulty is the dominant variance component, and pairing removes it entirely. This routinely halves the number of examples needed.

Graded outcomes. Binary success throws away information. A rubric scoring 0–4 on several dimensions has far lower variance per example than a coin flip, because a partially-correct answer is scored as partially correct rather than rounded to zero.

Evaluate the trajectory, not just the answer#

For agents, final-answer accuracy hides most of what you need to know. An agent that reaches the right answer after twenty-two tool calls, three of which were retries of a malformed request, is not the same product as one that gets there in four — even though the score is identical.

Dimensions worth scoring separately:

Efficiency deserves a specific warning: it is the metric that silently regresses. Success rate is what everyone watches, so a prompt change that adds four points of accuracy and forty percent to token cost gets shipped, and then repeats. Track cost per successful task, not cost, and not success.

Model-as-judge, with the failure modes handled#

Human grading does not scale to hundreds of examples per iteration. Model judges do, and they have well-documented biases you must design around.

JUDGE = """You are grading one response against a rubric. Score each dimension
independently. Do not reward length, confidence, or formatting.

<task>{task}</task>
<reference>{reference}</reference>
<response>{response}</response>

For each dimension, quote the specific span that justifies the score before
giving the score.

Dimensions:
- correctness (0-4): factually right, per the reference
- completeness (0-4): covers what was asked, nothing missing
- faithfulness (0-4): every claim traceable to provided material
- efficiency (0-4): no unnecessary work or padding

Return JSON: {{"correctness": {{"evidence": "...", "score": n}}, ...}}"""

Four things that make judges usable:

1. Evidence before score. Forcing a quotation before the number reduces drift and gives you something to audit when a score looks wrong.
2. Position randomization. In pairwise comparison, judges favour whichever response came first. Randomize order and, ideally, run both orders and keep only the agreements.
3. Calibration against humans. Grade 50–100 examples by hand, compute agreement with the judge, and report it. A judge with 0.6 correlation to human judgement is a rough signal; one at 0.9 can gate a release. You do not know which you have until you measure.
4. A different model as judge. Models prefer their own output. Using the same model to generate and grade puts a thumb on the scale in a direction you cannot see.

Report judge–human agreement alongside every judged metric. A number produced by an uncalibrated judge is not a measurement, it is a vibe with a decimal point.

Building the set#

Where the examples come from determines what the evaluation is actually measuring.

Production traces are the best source. Sample real requests, stratified by category, including the ones that failed. This is the only source that reflects your actual input distribution, and the distribution is usually stranger than anyone's intuition.

Hand-written adversarial cases for the things that must never happen: prompt injection in retrieved content, requests that should be refused, ambiguous inputs that should trigger a clarification rather than a guess. These are not sampled — they are curated, and they only grow.

Synthetic examples are useful for coverage of rare categories and dangerous for everything else, because they are generated from the same priors your system has and therefore systematically miss the same things.

Whatever the source, version the set with content hashes, record which set produced which reported number, and never edit an example in place. An evaluation set that changes silently is worse than none, because it produces confident comparisons between numbers that were measured differently.

Regression testing in CI#

Nightly or per-PR, on a fast subset:

GATES = {
    "outcome":       Gate(baseline=0.72, tolerance=-0.03),   # may not drop 3 pts
    "faithfulness":  Gate(baseline=0.91, tolerance=-0.01),   # nearly no slack
    "cost_per_task": Gate(baseline=0.043, tolerance=+0.15,   # +15% allowed
                          direction="lower_is_better"),
    "p95_latency_s": Gate(baseline=11.2, tolerance=+0.20),
}

def gate(results, gates=GATES) -> list[str]:
    failures = []
    for name, g in gates.items():
        value = results[name]
        if g.violated(value):
            failures.append(
                f"{name}: {value:.3f} vs baseline {g.baseline:.3f} "
                f"(tolerance {g.tolerance:+.0%}) — n={results['n']}, "
                f"CI ±{results[f'{name}_ci']:.3f}"
            )
    return failures

Printing the confidence interval next to every gate result is not decoration. It is what stops a team from chasing a "regression" that is inside the noise floor, which otherwise consumes a genuinely surprising fraction of engineering time.

What we would build first#

If a team has nothing, in order:

1. Trace capture. Every run persisted — prompts, tool calls, outputs, cost, timing, model version. Without this there is nothing to evaluate.
2. Fifty hand-graded examples from real traces, with a written rubric. Small enough to do in a day, and the rubric is the artefact that matters.
3. A judge calibrated against those fifty. Report the agreement.
4. Four hundred examples, judged, run per-PR with gates.
5. An adversarial set that only ever grows, run before every release.

Steps one and two are most of the value, and they involve no machine learning at all. The most common reason teams cannot tell whether a change helped is not that evaluation is hard — it is that nobody wrote down what "better" means before they started changing things.