When someone asks Google about a visa, the first thing they read is now usually Google's own answer, written from official pages and other people's. Nobody had measured whether those answers were right, so I did: 200 real questions, every answer captured word for word, and every fact in it checked against Immigration New Zealand's official pages.
- 200 real questions, typed as people type them
- 199 answered by an AI Overview
- 2,008 separate facts checked
- 74% of those facts hold up
- 27 of 200 answers with nothing wrong at all
Why it mattered
An AI Overview leaves no trace in a website's analytics. A click from it looks like any other Google click, and a reader who doesn't click leaves nothing at all. So the usual measures can't say what the public is being told, or whether it's right.
Immigration is where a wrong answer costs the most: a refused boarding, a missed deadline, a fee paid for an application that can't be accepted.
How I did it
- Real questions, not invented ones. I started from the 20,557 search terms in Immigration New Zealand's Search Console data, merged near-duplicates into 3,087 questions, kept the 923 about immigration that are asked often enough to matter, and drew 200, weighted towards the most-searched and the highest-stakes.
- Asked as the public would. Signed out of Google, on a New Zealand connection, with every question typed exactly as people typed it, spelling mistakes included. Every answer was saved word for word.
- Every fact checked. Each answer was broken into separate facts, about ten per answer, and each fact was looked up on the live official page, with the sentence that settles it recorded beside the verdict.
- Verdicts challenged. The checking was done with an AI model (Claude), quoting the official wording and link for every verdict. Anything marked flatly wrong then went to a second review briefed to defend Google's version. Only the ones that survived are called wrong.
What I found
Three quarters of what Google says about immigration holds up. The rest doesn't spread out thinly; it bunches together.
Because an answer is about ten facts and a reader can't tell which one is wrong, the whole answer is what counts. Only 27 of 200 answers had nothing wrong in them. Weighted by how often people search, just 5% of searches land on a clean answer.
An official link doesn't mean an official answer
89% of answers linked to immigration.govt.nz, so on the surface the official source owns this space. But only 37% of the 1,402 links Google showed went there. The rest went to immigration advisers, education agents, YouTube and Facebook. And 20 of the 21 answers with a plain factual error were linking to Immigration New Zealand while they made it. The link makes the answer look official, and makes a reader less likely to check.
The answers don't stay the same
I asked 20 of the questions three times on different days. The first source changed on 14 of them, and on 10 it switched between a government site and a commercial one. Only 46% of the facts appeared in all three answers. One question about character requirements was led by a different consultancy or law firm on each of the three days, and never by the official page.
Why it keeps happening
Google isn't inventing much. Nearly every error traced back to an official page that is correct but easy to misread: a limit in one sentence and its condition in the next, or the same figure published on two pages and updated at different times. The number travels; the condition that makes it true gets left behind.
The first of those comes from the most-searched question in the study. A passport that expires the week after the return flight passes Google's test and fails the real one, and the airline applies the real one at check-in.
What it changed
The report recommended three habits for anyone writing government content, on top of the content design guidance MBIE already has:
- Keep the condition in the same sentence as the number, so it's still true when a machine lifts that one sentence
- Put a date on anything that changes: fees, thresholds and wait times
- Publish each fact once and link to it from everywhere else, because duplicates drift and the AI picks one
It also made the case that the thing to watch has shifted, from how many people reach a page to whether what's said from it is right. The findings fed into MBIE's direction on AI-assisted search and its approach to generative engine optimisation (GEO).
- AI Overviews
- Answer accuracy
- GEO
- Search Console
- Claude
- Content design