Test it with the questions people really ask, check each answer fact by fact against the page it came from, and keep testing after launch. A good demo and a high satisfaction score don't tell you whether the answers are right; only checking them does.
Why a demo isn't evidence
Demos use questions chosen to work. Real questions are messier: misspelt, half-finished, phrased in people's own words, and often about the exception rather than the rule. They're also where a wrong answer costs the most.
Satisfaction ratings don't help much either. People rate an answer on whether it sounds helpful, and a confident wrong answer sounds very helpful.
1. Build a test set from real questions
Start with what people already ask you. Your site search logs, Google Search Console and the contact centre's top call reasons are all good sources. Pick a set that is:
- weighted towards the most common questions and the ones where a wrong answer hurts most, such as eligibility, fees, deadlines and anything legal;
- written the way people write, not tidied up;
- mixed with questions the tool should decline or hand to a person.
A first test set for one service doesn't need to be large. What matters is that the questions are real, that they lean towards the common and the high-stakes ones, and that you keep the set so you can run it again.
2. Check facts, not impressions
Split each answer into its separate facts and check each one against the official page, quoting the sentence that settles it. Then sort the facts:
- correct;
- correct but missing a condition that matters;
- not supported by any official source;
- contradicted by the official page.
Count whole answers as well as facts. An answer can be mostly right and still mislead, because a reader can't tell which of several facts is the wrong one. The share of answers with nothing wrong in them matters more than the share of facts that hold up.
If you use an AI model to help with the checking, give anything it marks as wrong a second review that's briefed to defend the answer. Only call it wrong if it survives that. Check a sample of the answers it marks as right too, or its misses stay invisible. A model checking a model isn't independent: a person who knows the rules makes the final call, starting with the answers that could do the most harm.
3. Look for the patterns
Sort each wrong answer by cause. Often it's the content: a number in one sentence and its condition in the next, the same figure on two pages, a fee with no date. Fixing those pages often fixes several answers at once. Other causes need other fixes: the tool retrieved the wrong page or an old copy, picked a third-party source over yours, or the model misread a page that was clear. More in why AI gives wrong answers about your agency.
4. Keep testing after launch
Answers change when content changes, when the model changes and sometimes for no visible reason. The same question can get a different answer, from a different source, on a different day. Run the same test set after every significant content or model change, and on a regular schedule.
Track a few numbers over time: the share of answers with nothing wrong, the number of errors on high-stakes questions, and how often the tool correctly declines.
5. Decide who owns what
- The business owner. Decides what a good answer is, and what error rate is acceptable for which questions.
- The content team. Fixes the pages behind wrong answers.
- The vendor or developers. Fix how the tool finds and uses sources.
The Public Service AI Framework asks agencies to adopt AI responsibly. For an AI tool that answers the public, testing its answers is where that becomes something you can show.
I help agencies set up answer testing, from the first test set to the routine checks after launch. See AI project consulting. If the question is what public AI tools already say about your services, the findability audit checks that.