Quick answer: A useful evaluation set contains situations that represent your real work, including examples where the correct action is to refuse or escalate. Test the entire workflow, not only whether an AI model writes a plausible sentence.
Build 40 meaningful cases
For a service desk, collect 20 ordinary inquiries, 8 unusual but legitimate cases, 6 inputs with missing or conflicting facts, and 6 adversarial or potentially sensitive cases. If these examples come from customer records, de-identify them and obtain any necessary approvals before using them in tests.
Make each expected result testable
A good expected output might say: “Classify as billing review; do not send; escalate to finance.” A vague result like “handle professionally” cannot be evaluated consistently. Include expected tool calls, allowed outputs and forbidden actions.
Score the parts separately
| Dimension | What to check |
|---|---|
| Correct routing | Did the input reach the right queue? |
| Factual grounding | Does every key claim match an approved source? |
| Safety | Did a risky case correctly stop? |
| Tool behavior | Were the only permitted actions executed? |
| Recovery | Can the team inspect and correct a failure? |
Avoid reporting one aggregate “accuracy” metric if failures have radically different costs. Missing an urgent account-security request is not equivalent to using a slightly awkward greeting.
Test changes and regressions
Freeze a baseline set, record model and prompt versions, and re-run after changing prompts, document sources, permissions or workflow logic. Include one person not involved in authoring the prompt to review ambiguous examples.
FAQ
Is a 40-case test enough for production?
It is an entry-level smoke test, not a guarantee of reliability. Expand testing based on risk, input diversity and real failures before increasing autonomy.
References and further reading
These references provide context and official guidance. The sample workflows above are illustrative designs, not vendor performance tests.
Published by: FlowHarbor Editorial Desk. Read our sourcing and review policy.