Someone tells you their agent scores 84 percent. The 16 percent is where the people are. Somebody asked for something and did not get it, and if you cannot say what happened to them, you cannot make the product better for them. That is what an eval is for. Not a grade, a list of experiences to go fix.

An eval is a pile of realistic requests you run your system on and grade. Change something, run it again, see if the score moved.

You need one because ordinary testing does not work here. Normally you put something in, expect exactly one thing out, and if it does not match, something broke. AI is not like that. Two different answers can both be fine. So you stop asking whether it is correct and start asking how often it is good enough.

Good enough is not a fact about the system. Somebody decides it. That is where all the trouble is, and it is what I spend my time on: I work on eval generation and rubric design at Thoughtful Labs, which means producing the cases that test a system and writing the rules that decide what passes.

The score is the easy half

Running the tests is a script. Deciding what counts as passing is the work.

I wrote eight agent transcripts and graded them against a three line rubric. Pass, recoverable, fail. Five of them the rubric settles on its own. Three it does not.

Five of the eight transcripts the rubric settles on its own. On three of them it says nothing, and those three are the useful ones.eight transcripts, one three line rubricRefund one day latethe rubric decides itFake citationthe rubric decides itEdited the testthe rubric decides itRefused a self-serve answerthe rubric decides itErrored, then caught itthe rubric decides itRight number, wrong methodthe rubric is silentStopped to askthe rubric is silentNine of twelvethe rubric is silent
The five the rubric already settles tell you almost nothing. You knew those before you started. The three at the bottom are the whole return on the exercise.
  • Right number, wrong methodThe query timed out. The agent repeated a figure the user had mentioned earlier, and it happened to be correct.Does a right answer reached the wrong way count?
  • Stopped to askBooking the meeting would have put a colleague in a call at two in the morning, so the agent came back with options instead.Is stopping to ask a success or a failure?
  • Nine of twelveIt migrated nine call sites, skipped three that needed a product decision, and said exactly which three and why.Is there a grade between done and not done?

None of those three have correct answers. They have answers your team picks and writes down. Until somebody does, two people grading the same transcript land in different places and both of them are right.

Which means 84 percent is not a property of your system. Swap the grader and the number moves while the product sits there unchanged.

The test nobody writes

One more, because it is the case I think is most often missing entirely.

A user asks how to cancel their subscription. The agent looks it up, finds that cancelling is a setting the user can already reach, and then declines to tell them, citing account security. Nothing was protected. The user loses an afternoon to a phone call for something the product supports.

That is a failure, and it is a failure almost no eval will catch, because nobody writes test cases where the right answer was withheld. Refusals feel safe. They are easy to defend in a review. Every incentive in a grading process pushes toward counting them as passes.

Unhelpfulness is not a safe default. It is a failure that happens to be comfortable, and if your eval set only contains cases where the model might do too much, you have built an instrument that can only find one of the two ways to be wrong.

What to do about it

Take ten transcripts. Have two people grade them separately. Throw out every case you agreed on and spend the whole meeting on the rest. Every disagreement becomes a sentence you add to the rubric. That is the entire method and it costs an afternoon.

I am building the eight transcripts into a tool that does this for you, called The Grading Room. It hands you the transcripts, takes your grades, and shows you where you and the reference part ways. If you want to be one of the people it learns from, tell me.

One caveat worth stating plainly: that those three split is a prediction, not a result. Nobody outside me has graded them yet. If it turns out everyone grades all eight the same way, I am wrong about this, and I would want to know.