Reef TRHThe room is full, the screen is up, and the AI has behaved perfectly through three weeks of demos. Someone reaches for the switch that puts it in front of real customers, and a quieter voice asks the only question that matters: how do we actually know it's right? Nobody has a number. They have a good feeling.
A good feeling is not evidence. You do not learn whether an AI is right from a demo; you learn it from evaluation, a deliberate test of the system against cases where you already know the correct answer, run before it ever meets a customer and again continuously after. A demo shows the AI can be right once. An evaluation shows how often it is right, and exactly where it is not.
The demo is the most dangerous evidence you have
A demo is a highlight reel. It runs the happy path on friendly inputs chosen by the person who built it. Production is the opposite: the Tuesday-afternoon caseload, the scanned document sitting slightly crooked, the customer who describes the problem wrong. Judging an AI by its demo is like judging a car by the showroom floor. It is the applause-in-the-room, silence-in-production gap that strands so many pilots, and a large part of why most never reach production.
Write the answer key before you trust the answer
Evaluation needs a gold set: a few hundred real cases pulled from your own work, each with the correct outcome a human has already agreed. Not the easy ones, the representative ones, deliberately including the hard, rare and sensitive cases where a wrong answer is expensive. Then you run the AI across the set and measure four things:
- How often it is right across the whole set.
- How often it is right on the hard slice, the awkward cases where demos quietly fail.
- What "wrong" actually looks like, so you understand the failure, not just the rate.
- Whether it will say "I'm not sure" instead of inventing an answer with confidence.
Only now do you have a number instead of a vibe, and something you can defend to anyone who asks.
Set a floor you will not cross
Below a set confidence bar, the AI does not answer on its own; it hands the case to a person. One insurer we worked with put a hard 85% confidence floor under every answer, with a human fallback beneath it. The system is allowed to be unsure. It is never allowed to guess in silence.
The evaluation is what tells you where that floor belongs. This is the move that turns "it usually works" into "it works, and here is precisely what happens when it does not". It is the difference between an AI you can vouch for and one you can prove.
Evaluation is a habit, not a milestone
The world moves after launch. Models drift, your data shifts, a supplier changes a format, and yesterday's number goes stale. A one-time evaluation is a photograph; production needs a live feed. The systems that survive re-run the check continuously and raise a flag the moment accuracy slips, the same discipline as fact-check agents that block any claim they cannot trace to a trusted source. If you only measured once, you do not know whether your AI is right today. You know it was right on the day you looked. Which checks to run before that first launch is its own list, and we have written it down.
None of this is glamorous, and none of it shows up in a demo. That is the point. "How do you know it's right?" sounds like a technical question, but it is really a question about trust: you cannot hand a customer, or a regulator, an AI you can only vouch for with a good feeling. Build the answer key first. The confidence comes after.
Common questions
What's the difference between a demo and an evaluation?
A demo shows the AI can be right once, on inputs someone chose. An evaluation measures how often it is right across hundreds of real cases, including the ones that break demos.
How big does an evaluation set need to be?
Big enough to cover your real range, hard and rare cases included. A few hundred well-chosen, human-verified examples beat thousands of easy ones.
Do we need clean data before we can evaluate an AI?
No. The messy cases are the whole point, because they are where AI fails in production. Structuring the evaluation set is part of the work, not a prerequisite you must finish first.