.png)
Most AI failures are found by customers, not teams. This Guide shows how to build a five-day evaluation set that catches them first, before and after you ship.
On this page
Evaluating AI Systems Before You Ship Them
Most teams discover whether an AI system works after someone has already trusted the wrong answer.
A customer receives it. A colleague acts on it. A manager asks why nobody caught it before launch.
The solution is not an enormous testing programme. It is a small, honest evaluation set: a fixed collection of representative inputs, agreed expectations, and known failure conditions that you run before every meaningful change and continue running after release.
It may be the cheapest insurance in the entire AI project. A useful first version can be built in five working days.
What is an AI evaluation set?
An evaluation set is a stable group of examples used to test whether an AI system behaves as intended.
Each example includes:
- A realistic input
- The expected answer or acceptable outcome
- The mistakes that would make the output unacceptable
- The cases that must be escalated to a person
Unlike a polished demo set, it should represent the conditions the system will encounter in normal use. That includes vague questions, incomplete information, unusual phrasing, conflicting context, and requests the system should refuse or hand over.
The point is not to prove that the AI can succeed. The point is to find out where it cannot be trusted yet.
Day 1: Define what “good” means
Before collecting examples, write one clear sentence describing the outcome the system must produce and who is accountable for it.
“The assistant should be helpful” is not a testable standard.
A useful definition might specify:
- The percentage of answers that must be factually correct
- The types of mistakes the business cannot tolerate
- The maximum acceptable response time
- The conditions that require human review
- The person responsible for approving release
This definition becomes the basis for every decision that follows. Without it, teams evaluate outputs against instinct. Instinct changes depending on who is reviewing the answer and how close the launch date is.
If the team cannot agree on what good means, the system is not ready to be evaluated, let alone shipped.
Day 2: Collect real examples, including the awkward ones
Start with 50 to 100 examples drawn from the messiest representative source available.
Use real support tickets, documents, searches, customer questions, internal requests, or workflow records. Remove sensitive information, but preserve the ambiguity and inconsistency that made the original example difficult.
Your set should contain common requests as well as edge cases:
- Incomplete questions
- Misspellings and shorthand
- Conflicting instructions
- Missing source information
- Requests outside the system’s scope
- Inputs that should trigger a refusal or human handover
The clean examples used in a demonstration show that the system can work under friendly conditions. Real examples show what will happen after release.
Day 3: Decide what counts as correct before seeing the output
For each example, document the expected outcome before running it through the system.
Depending on the use case, this may be one correct answer, a range of acceptable responses, a required source, or a decision to escalate. Also record what would make an answer wrong.
Doing this in advance prevents hindsight bias. When reviewers see a fluent answer, they often adjust the standard and accept something that only “sounds about right.” AI systems make this especially easy because a polished response can disguise a factual or procedural error.
Create a separate escalation category for cases that should not be decided by the model. This may include decisions involving payments, legal exposure, regulated information, safety, or commitments made to customers.
The model does not need to answer everything. It does need to recognise when it should not answer alone.
Day 4: Run the evaluation and examine every failure
Run the complete set and score each output against the agreed standard.
A simple first scoring model is enough:
- Correct: The output meets the required standard.
- Acceptable: The output is usable but contains a minor issue that does not change the decision or outcome.
- Wrong: The output is misleading, unsupported, unsafe, or unsuitable for use.
- Escalation failure: The system answered when it should have involved a person.
Do not reduce the results to one pass-rate percentage too quickly.
A 90% score can describe two very different systems. One may have ten minor style problems. Another may fail every time a customer asks about refunds or contractual terms. The average conceals the difference.
Read the failures together. Group them by cause. Look for patterns such as missing context, weak retrieval, ambiguous instructions, unsupported confidence, incorrect tool use, or an escalation rule that is not being followed.
The purpose of the review is not to defend the score. It is to understand the risk hidden inside it.
Day 5: Set a release threshold and a continuing cadence
A one-time evaluation tells you how the system performed on one day.
Models change. Prompts change. Connected data changes. Customer behaviour changes. Even when the application code stays the same, the quality of the overall system can drift.
Before closing the week, agree on four operating rules:
- When the evaluation will run, such as weekly and before every meaningful release
- Who owns reviewing and signing off the results
- Which score or failure category will stop a release
- How new production failures will be added to the evaluation set
That final rule is essential. Every meaningful failure discovered after launch should become a permanent test. Otherwise, the organisation learns from the incident once while the system remains free to repeat it.
An evaluation set nobody reruns is a photograph being mistaken for a live feed.
What a week of evaluation gives you
At the end of five days, you should have:
- A shared definition of acceptable performance
- A representative set of real inputs
- Expected outcomes agreed before testing
- A visible record of important failure modes
- Clear human-escalation rules
- A release threshold and named owner
- A repeatable evaluation cadence
None of this requires an elaborate evaluation platform at the beginning. A carefully maintained spreadsheet and a disciplined review process can be more valuable than sophisticated tooling built around weak examples.
The quality of the set matters more than the complexity of the system running it.
Build the evidence before you ship the confidence
Traditional software often fails visibly. It crashes, returns an error, or produces a broken screen. AI systems can fail in complete sentences.
That makes evaluation a product discipline, not a final QA activity. Teams that skip it often learn about reliability from a customer, a regulator, or a board member. Teams that build it establish what the system can be trusted to do before asking anyone else to trust it.
Build the evaluation set before you ship. Keep running it after you do.
Download this guide to get started AI project failure and production readiness.pdf
Frequently asked questions
What is an AI evaluation set?
An evaluation set is a stable, representative group of examples used to test whether an AI system behaves as intended. Each example pairs a realistic input with the expected answer, the mistakes that would make it unacceptable, and the cases that should be escalated to a person instead of answered by the model.
How is an evaluation set different from a demo set?
A demo set is chosen to show the system working under friendly conditions. An evaluation set is built from the messy, ambiguous, awkward inputs the system will actually face in production, including incomplete questions, conflicting instructions, and requests it should refuse or hand over.
How many examples do I need to start?
Fifty to one hundred real examples is enough for a useful first version, drawn from actual support tickets, documents, or customer questions rather than curated demo cases, preserving the ambiguity and inconsistency of the originals.
Why shouldn't I decide what counts as correct after seeing the AI's answer?
A fluent, polished response can make reviewers unconsciously lower the bar and accept something that only sounds right. Documenting the expected outcome before running the evaluation prevents that hindsight bias.
Why is a single pass-rate percentage risky to rely on?
A 90% score can describe two very different systems: one with ten minor style issues, or one that fails every time a customer asks about refunds. Grouping failures by cause reveals the risk hiding inside the average.
How often should an evaluation set be run after launch?
At minimum, before every meaningful release, and on a regular cadence such as weekly, since models, prompts, connected data, and customer behaviour can all drift even when the application code stays the same.
What should happen when the system fails in production?
Every meaningful failure discovered after launch should be added back into the evaluation set as a permanent test case, so the organisation doesn't just learn from an incident once while the system stays free to repeat it.
About the author:
Ahana Roy
Content Marketing Manager
A writer at heart and a marketer by choice, Ahana heads content and social media at SDTC Digital, bringing an instinct for language and a sharp eye for what moves people. Working across the blog and social channels every day, she sees firsthand which stories earn attention and which get lost in the feed.
.png)
.png)
.png)
