Test it with the questions people really ask, check each answer fact by fact against the page it came from, and keep testing after launch. A good demo and a high satisfaction score don't tell you whether the answers are right; only checking them does.
Why a demo isn't evidence
Demos use questions chosen to work. Real questions are messier: misspelt, half-finished, phrased in people's own words, and often about the exception rather than the rule. They're also where a wrong answer costs the most.
Satisfaction ratings don't help much either. People rate an answer on whether it sounds helpful, and a confident wrong answer sounds very helpful.
1. Build a test set from real questions
Start with what people already ask you. Your site search logs, Google Search Console and the contact centre's top call reasons are all good sources. Pick a set that is:
- weighted towards the most common questions and the ones where a wrong answer hurts most, such as eligibility, fees, deadlines and anything legal;
- written the way people write, not tidied up;
- mixed with questions the tool should decline or hand to a person.
A first test set for one service doesn't need to be large. What matters is that the questions are real, that they lean towards the common and the high-stakes ones, and that you keep the set so you can run it again.
| Question | Tool | Official answer | What it said | Result |
|---|---|---|---|---|
| My recycling wasn't collected today, what do I do? | Google AI Mode | Not collected by 3pm? Call 04 499 4444 | Call before 3pm; after 3pm, wait for next week | Wrong |
| How much to register a desexed dog in Wellington? | ChatGPT | $148, or $222 on or after 1 August | Same figures, cited from a pet site | Unofficial source |
| How much does the government put into my KiwiSaver? | Google AI Overview | 25c per $1, up to $260.72 a year | Right, cited from an insurance broker | Unofficial source |
| The same KiwiSaver question | ChatGPT | 25c per $1, up to $260.72 a year | Right, cites Inland Revenue | Correct |
| The same KiwiSaver question | Claude | 25c per $1, up to $260.72 a year | Right, mostly cites KiwiSaver providers | Unofficial source |
| When are Wellington City Council rates due? | Google AI Overview | 1 September, 1 December, 1 March, 1 June | All four dates, cites the council | Correct |
2. Check facts, not impressions
Split each answer into its separate facts and check each one against the official page, quoting the sentence that settles it. Then sort the facts:
- correct;
- correct but missing a condition that matters;
- not supported by any official source;
- contradicted by the official page.
Count whole answers as well as facts. An answer can be mostly right and still mislead, because a reader can't tell which of several facts is the wrong one. The share of answers with nothing wrong in them matters more than the share of facts that hold up.
If you use an AI model to help with the checking, give anything it marks as wrong a second review that's briefed to defend the answer. Only call it wrong if it survives that. Check a sample of the answers it marks as right too, or its misses stay invisible. A model checking a model isn't independent: a person who knows the rules makes the final call, starting with the answers that could do the most harm.
3. Look for the patterns
Sort each wrong answer by cause. Often it's the content: a number in one sentence and its condition in the next, the same figure on two pages, a fee with no date. Fixing those pages often fixes several answers at once. Other causes need other fixes: the tool retrieved the wrong page or an old copy, picked a third-party source over yours, or the model misread a page that was clear. More in why AI gives wrong answers about your agency.
4. Keep testing after launch
Answers change when content changes, when the model changes and sometimes for no visible reason. The same question can get a different answer, from a different source, on a different day. Run the same test set after every significant content or model change, and on a regular schedule.
Track a few numbers over time: the share of answers with nothing wrong, the number of errors on high-stakes questions, and how often the tool correctly declines.
5. Decide who owns what
- The business owner. Decides what a good answer is, and what error rate is acceptable for which questions.
- The content team. Fixes the pages behind wrong answers.
- The vendor or developers. Fix how the tool finds and uses sources.
The Public Service AI Framework asks agencies to adopt AI responsibly. For an AI tool that answers the public, testing its answers is where that becomes something you can show.
I help agencies set up answer testing, from the first test set to the routine checks after launch. See AI project consulting. If the question is what public AI tools already say about your services, the findability audit checks that.