# Checking AI Overviews for accuracy https://miromiro.co.nz/projects/ai-overviews-accuracy/ A method case study from MBIE: checking Google's AI Overviews on immigration questions against official pages, grading answers, and three writing habits. --- Project # Checking AI Overviews for accuracy By Miro Bzduch · October 2026 When someone asks Google about a visa, the first thing they often read is Google's own answer. At MBIE, I checked those answers, fact by fact, against Immigration New Zealand's official pages. This page sets out the method and the main lesson for content owners. The checks were internal MBIE work, so the questions, figures and findings aren't published here. Where MBIE, 2026 What Google's AI Overviews on immigration questions, checked against Immigration New Zealand's pages My part The method and the checks Results Internal to MBIE, not published ## Why it matters [Image: Google results for a question about how long a passport must be valid for a New Zealand visitor visa, led by an AI Overview citing Immigration New Zealand] What a signed-out search shows for a passport question (October 2026): Google's AI Overview comes first, above Immigration New Zealand's own page. When someone reads an AI Overview and doesn't click through, the website's analytics can't show what they read, so the usual measures can't say what the public is being told. And in immigration a wrong answer is expensive: a refused boarding, a missed deadline, a fee paid for an application that can't be accepted. The answer also links to the official site, so it looks official, whether or not it matches what the site says. ## How I did it - Real questions. Written the way people type them, typos and all, leaning towards common and high-stakes ones. - A clean search. Signed out, on a New Zealand connection, with every answer saved word for word. - One fact at a time. Each answer split into its separate facts, and each checked against the live official page, quoting the sentence that settles it. - A second opinion. Claude did the first pass. Anything marked wrong went to a second review briefed to defend Google's version, and only what survived is called wrong. Both passes use a model, so neither is independent verification: they narrow down what a person who knows the rules has to check. ## How an answer is graded A single answer can be mostly right and still mislead. Here's how one is taken apart, using a made-up council question so the method is clear. Made-up example: "can i renew my dog rego online" AI Overview Yes, you can renew your dog registration online.1 The fee is $150.2 Renewals are due by 31 July.3 You'll get 10% off for paying early.4 The late fee is $50.5 - Correct The council page says you can renew online. - Leaves out a condition $150 is the fee for desexed dogs. Everyone else pays more. - Correct The due date matches. - No official source Nothing on the council's site mentions an early discount. - Official page says otherwise The late fee is half the registration fee, not a flat $50. Two of the five facts are correct. One leaves out a condition, one has no official source, and one is contradicted by the official page. Someone acting on this answer could pay the wrong amount twice. That's why the whole answer counts, not the share of facts that are right. "No official source" isn't the same as wrong. The claim may be true, but nothing official backs it, so nobody should rely on it. ## The main theme: the number travels, the condition stays behind In the answers I checked, Google wasn't inventing much. Where an answer went wrong, the official page was often correct but easy to misread: a limit in one sentence and its condition in the next, or the same figure on two pages. Made-up example: how a condition gets lost The official page Dog registration costs $150. This fee applies to desexed dogs. → The AI answer The fee is $150. Both sentences are true on their own. Read one at a time, as an AI tool often does, the first one is wrong for most dog owners. ## What content owners can do Three habits help any team writing government content. Each is a small edit. Before After "Registration costs $150. This applies to desexed dogs." "Registration costs $150 for a desexed dog." "Processing takes about 4 weeks." "Processing takes about 4 weeks (as of July 2026)." The fee written out on five different pages The fee on one page, with the others linking to it - Keep the condition in the same sentence as the number - Put a date on anything that changes, such as fees and wait times - Publish each fact once and link to it, because duplicates drift The thing to watch is no longer only how many people reach a page, but whether what's said from it is right. - AI Overviews - Answer accuracy - GEO - Claude - Content design Related: [Showing up in AI answers](https://miromiro.co.nz/findability-audit/generative-engine-optimisation/) · [Smarter site search](https://miromiro.co.nz/ai-consulting/ai-search/) ## Talk about your project I [consult on AI projects](https://miromiro.co.nz/ai-consulting/), from where AI is worth using to testing the answers, and run [findability audits](https://miromiro.co.nz/findability-audit/), for government agencies and Crown entities. [Get in touch](https://miromiro.co.nz/contact/)