An agent that works once is easy to build. You run it, the output looks right, you call it done. Nate Herk, who runs an AI automation agency and has spent five thousand hours building these systems, calls that the trap: one run proves the agent worked one time on one input. Before it touches real customers, you want a number, the share of a hundred real runs it gets right. This guide is the cheap way to get that number.
01One run tells you almost nothing
The models inside an agent are non-deterministic: ask the same model the same thing twice and the answers differ. Usually only the wording changes. Sometimes the answer itself changes. And when you swap the model for a newer one, the whole system reads its instructions differently.
“you really have no idea what your success rate is going to be across 100 real runs”
Nate Herk, “5000 Hours of Building AI in Just 17 Minutes” · 9:08Loop the agent until the work passes handles one side of this: a single run that checks its own output before it stops. The score handles the other side: how often those runs succeed across many inputs. You need both, because a run that passes its own checks today says nothing about the ninety-nine other inputs it will meet this month.
02Collect known good answers
The measuring tool is a scoring set: real examples of the task, each paired with an answer a person already judged good. Herk’s agency built one for a support agent that had to look up the customer, search the database, and write a reply. They collected 500 replies humans had written and approved, and that became the standard the agent was measured against. He calls this a golden data set; the name matters less than where the answers come from.
The answers come from work already done. Past tickets and the replies that closed them. Past reports and the version that shipped. Twenty examples are enough to start, and the set gets more trustworthy as it grows.
03Grade by script or by judge
Scoring a run means comparing the agent’s output to the known good answer. Who does the comparing depends on the check:
| The check is | Grade it with |
|---|---|
| Objective (the number matched, the format is valid, the fields are present) | A script |
| Judgment (does this reply resolve the ticket the way the approved one did) | A judging model |
A judging model is a second model handed the agent’s output, the known good answer, and the question “does this meet the bar”. The judge compares against an answer a person approved, so the standard comes from a person and the agent never grades its own work. The design rule for both graders is the same one Herk uses: write down how a human reviewer would check this work, then automate that exact review. If your reviewer would test the number, script it. If your reviewer would read both replies and compare, that is the judge’s job.
04Rerun the set after every change
Once the set exists, every change to the agent gets the same treatment: make the change, run all the examples, read the score. A new prompt line, a different tool setup, a model swap, each becomes a yes or no instead of a feeling.
“You can't just base it on a gut feeling. You have to actually prove it.”
Nate Herk, “5000 Hours of Building AI in Just 17 Minutes” · 10:21The reason to insist on the rerun: changes you are sure will help sometimes drop the score, and you would rather learn that from your own testing than from customers. The score is also what tells you the agent is ready to move from testing into real work, and it makes other decisions cheap. Recheck your model choices each release asks you to re-test models as new ones ship; with a scoring set, that recheck is one rerun. The evaluator-separate-from-worker rule in Automate the step that slows you down is the same idea for a single piece of handed-over work; the scoring set makes that evaluator repeatable.
Further reading
- Nate Herk · 5000 Hours of Building AI in Just 17 Minutes, the source this guide draws on
- Anthropic · Create strong empirical evaluations, the vendor’s own guide to building scoring sets