Key takeaways
- An evaluation set is a collection of realistic test cases with known good outcomes that you re-run after every change.
- Start with 50 to 100 real questions, include hard and adversarial cases, and review failures by hand.
- Score what matters to the business: correct answers, correct actions, use of the right sources and safe behaviour.
- Automated scoring, including a second model acting as a judge, is useful but should be checked against human review.
- Keep evaluating in production with feedback buttons, sampled reviews and alerts for unusual behaviour.
Why agents need tests
Traditional software behaves the same way every time. AI agents do not: the same question can produce different answers, and a change to a prompt, a model or a tool can improve some cases while quietly breaking others. Without a way to measure, teams rely on a handful of demos, which hides problems until customers find them.
An evaluation set, often shortened to "evals", turns quality into a number you can track.
Step 1: collect real cases
Start from reality. Pull real questions from support tickets, chat logs, emails or sales calls. Aim for 50 to 100 to begin, and grow over time. Include:
- Common requests that must always work.
- Edge cases and ambiguous questions.
- Requests the agent should refuse or hand to a person.
- Adversarial inputs, such as attempts to override instructions.
- Questions whose answer changed recently, to test freshness.
Step 2: define what good looks like
For each case, write down the expected outcome. Depending on the task, that might be a correct answer, the right source cited, the right tool called with the right parameters, or a correct refusal. Be specific enough that two reviewers would agree on whether the agent passed.
Step 3: score the results
| Method | Best for | Watch out for |
|---|---|---|
| Exact checks | Tool calls, structured outputs, numbers | Too rigid for free text |
| Human review | Tone, helpfulness, judgement | Slow and expensive at scale |
| Model-as-judge | Large-scale scoring of free text | Needs calibrating against human reviewers |
Most teams combine all three: exact checks wherever possible, a model judge for volume and regular human review to keep the judge honest.
Step 4: make it part of every change
Run the full set whenever you change a prompt, model, tool or knowledge source, and compare against the previous run. Track overall pass rate, but also look at categories, because an average can hide a serious regression in one area. Treat a drop in a critical category as a blocker.
Step 5: keep watching in production
- Add thumbs up or down and a way to flag a bad answer.
- Review a random sample of conversations every week.
- Turn every real failure into a new test case.
- Alert on unusual patterns, such as sudden refusals, long loops or cost spikes.
For agents that retrieve from your documents, also measure retrieval separately: did the right passage come back? Our explainer on RAG covers this. If you want help setting up evaluations for your own agent, talk to our AI agent development team.
Qodexify is a remote-first team of senior engineers building web apps, AI agents, workflow automation and websites. Learn more about our AI agent development service or talk to an engineer.
