Eval Lab

An AI demo can look pretty convincing. Give it a friendly example, get a good answer, and it is tempting to move on. But what happens when the request is unclear? When the answer needs information the system does not have? When the right response is to stop and ask a person?

Eval Lab is my attempt to make those questions easier to work with. The point is to help someone turn an idea for an AI product into a set of examples they can inspect. What should the system receive, what should a good answer do, and what would count as a mistake?

You should not need to know the vocabulary of evaluations before you can start. The current app opens with a welcome page and asks one question at a time. What should the AI do? Who is it helping? What would a good answer look like? What should it avoid? Context and things to avoid are optional, and there are examples if you get stuck.

Before anything is generated, you can read the answers together and change them. That is a fairly simple decision, but it matters here. If the description is wrong, the generated cases can be perfectly coherent tests of the wrong thing.

Behind the scenes, those answers are formatted into the generator’s existing inputs. The wording is preserved apart from trimming whitespace, and the answer about what to avoid is labeled and added to the context. There is no extra model call quietly rewriting every answer. The model is used to generate the cases after the person has had a chance to review the brief.

The cases are then editable. You can question the expected answer, change the checks, and export the set as CSV or JSON. I want the work to be useful outside the interface too. Generating a list is only a starting point; the person building the product still has to decide whether the examples make sense.

The first live set used a fictional library borrowing task. Its three cases covered a normal request, the borrowing limits, and a request that exceeded them. They also made assumptions about availability and membership. Those assumptions are exactly the kind of thing someone needs to catch before treating generated cases as a standard.

A useful set should help someone explain what their product is supposed to do, including the awkward cases. It also needs to leave room for the person to say, “That is not what I meant,” and fix it. That is why reviewing the brief and editing the cases are part of the flow, rather than treating the generated answer as finished.

A generated test set is not proof that an AI product works. It gives you something concrete to start checking. Whether those examples cover the failures that matter for a particular product is still a question for the person building it.