Your eval set is your spec — write it before the prompt
Why the most expensive AI mistake we see at customer engagements is teams tuning prompts before they've written the regression test that defines 'right'.
TLDR
Why the most expensive AI mistake we see at customer engagements is teams tuning prompts before they've written the regression test that defines 'right'.
- Why the most expensive AI mistake we see at customer engagements is teams tuning prompts before they've written the regression test that defines 'right'.
The mistake everyone makes
You have a new LLM feature. You write a prompt. You test it manually on five examples. It looks good. You ship. Three weeks later, a user screenshots a response that is confidently wrong, and you're back to prompt-tweaking by intuition with no idea whether your changes are helping or just moving the failure mode.
This cycle is the single most expensive pattern we see in AI product teams, and it's entirely avoidable. The fix is not better prompts. It's writing the eval set first.
What an eval set actually is
An eval set is a collection of (input, expected output) pairs — or for non-deterministic outputs, (input, assertion function) pairs — that defines what "correct" means for your feature.
Before you write your first prompt, you should be able to answer: "How will I know if this is working?" If the answer is "I'll try it and see," you don't have a spec. You have a hope.
A minimal eval set for a document summariser might look like this:
input: SEC 10-K filing, 40 pages
assertion: summary under 200 words
assertion: mentions revenue, net income, and risk factors
assertion: no hallucinated figures not present in source
That's it. Three assertions, one input. But now you have something you can run against every prompt variant, every model upgrade, and every change to your retrieval pipeline.
Why teams skip this step
The honest reason is that writing evals feels like work that delays shipping. And in the short run, it does. A minimal eval set for a new feature takes two to four hours to write properly.
But consider the alternative. Every manual test you run is a one-shot, non-reproducible experiment. You cannot compare prompt A to prompt B with statistical confidence. You cannot know whether switching from GPT-4o to Claude costs you quality or gains it. You cannot catch regressions before they reach users.
The two-to-four hours you spend writing evals buys back every hour you would otherwise spend debugging production issues by intuition.
Anatomy of a good eval
A good eval has three properties: it's specific, it's reproducible, and it covers the failure modes you actually care about.
Specific means the assertion tests one thing. "The response is good" is not an assertion. "The response does not contain the word 'certainly'" is. "The extracted date is within 30 days of the document date" is.
Reproducible means you can run it in a script, without a human in the loop, in under five minutes. If your eval requires a human to judge quality, it's a spot-check, not an eval. Use LLM-as-judge patterns sparingly and always with a calibration set to validate the judge itself.
Coverage means your evals include the edge cases and adversarial inputs that matter for your use case, not just the happy path. A customer service bot eval set that only contains polite, well-formed questions will pass with flying colours and fail on the first rude user.
The eval-first workflow
- Define the feature's success criteria in plain language before you write any code.
- Translate each criterion into one or more executable assertions.
- Collect 20–50 representative inputs, including at least five edge cases and five adversarial examples.
- Run baseline: zero-shot against the model with no custom prompt. Record scores.
- Iterate on your prompt against the eval set, not against manual testing.
- Gate deployment on a minimum score threshold, not on "it felt good in the demo."
Evals as a living document
Your eval set is a specification that evolves with your product. Every production failure should add a new eval case. Every model upgrade should run against the full set before deployment. Every prompt change should be measured against the baseline.
The teams we see shipping reliable AI products consistently treat their eval sets as first-class engineering artifacts — version-controlled, reviewed in PRs, and maintained alongside the feature code. The teams stuck in the prompt-tweak-and-pray cycle consistently treat evals as an afterthought.
The eval set is not documentation. It is the spec.