Decide what it must do before you look at vendors, then test the shortlist on your own content with the questions your users really ask, and compare the answers, not the feature lists. A demo shows what a platform can do on a good day. A test on your content shows what it will do on yours.
Start with what it must do, not who sells it
Before any vendor meeting, write down:
- who it's for, and the questions it should answer;
- what it must not answer, and when it should hand over to a person;
- what a wrong answer would cost, question by question;
- which content it may draw from, and how quickly a changed page must change the answers.
Agree them with the people who will own and support the tool after launch, not only the project team, because they are the ones who will live with the choice.
Written well, the same requirements work whether you build in-house, add AI to a platform you already license, or buy something new. That keeps the comparison fair.
Shortlist across the options
Many search and content management platforms now offer AI features, so the first option to check is often the one you already pay for. Put it on the shortlist next to a specialist provider and a large platform, and judge them the same way.
Run a test on your own content
Give every shortlisted platform the same content and the same set of real questions, taken from your search logs, Search Console and contact centre. Include the hard ones: misspellings, exceptions, questions with conditions, and questions it should decline.
Then score the answers without knowing which platform gave them, fact by fact against your pages, the way I describe in how to test whether your AI gives correct answers. Check that each source it cites is the right page. A link to your home page doesn't count.
Keep the test small enough to repeat. A few dozen well-chosen questions, scored carefully, tell you more than hundreds scored in a hurry, and you can run the same set again when a platform releases a new model or you change your content.
| Measure | Platform ASearch vendor add-on | Platform BCloud AI service | Platform CChatbot product |
|---|---|---|---|
| Answers | |||
| Correct and complete | 41 / 60 | 52 / 60 | 47 / 60 |
| Links to the page it used | 60 / 60 | 58 / 60 | 31 / 60 |
| Says so when the pages don't cover it | 4 / 8 | 8 / 8 | 2 / 8 |
| Uses a page updated yesterday | Yes | Yes | Weekly sync |
| Running it | |||
| Median response time | 1.8 s | 3.1 s | 2.4 s |
| Data stays in Australia or New Zealand | Yes | Yes | US only |
| Your team can change the prompt | No | Yes | Via vendor |
ReadPlatform B answers best and keeps you in control. Platform A is fastest and always links its source, but your team can't change how it answers.
Look past the answers
Two platforms with similar answers can still be very different to run. Ask about:
- Control over sources. Can you exclude pages, and see which page each answer came from?
- Freshness. How long after you change a page do the answers change?
- Logs. Can you export questions and answers to review and re-test?
- Privacy and data. Where are questions stored, for how long, and are they used for training?
- Accessibility. Does the interface meet the Web Accessibility Standard your agency is held to?
- Cost at your volume. What does it cost per month at your real number of questions, not the pilot's?
Red flags
- The vendor only wants to demo its own questions.
- Answers can't show their sources.
- You can't export the questions and answers.
- There's no way to stop it answering from content you don't want used.
Write the decision down
Record what you tested, the scores, and why you chose what you chose. It makes the procurement defensible, and when the next model or platform comes along, you can run the same test again instead of starting over.
At MBIE I was part of a team that compared AI and enterprise search platforms, and I wrote requirements for AI-assisted search. More in the MBIE findability work.
I help agencies write requirements and run platform tests like this. See AI project consulting. If the question is what public AI tools already say about your services, the findability audit checks that.