Why AI answers change every time, and how to report AI visibility honestly

An AI visibility tool can be accurate about what it recorded and still mislead about what it means. Answer engines rarely give the same answer twice, the API a tool asks is not the app your client's buyer uses, and the sources engines cite drift from week to week. None of that makes measurement pointless. It changes what a defensible figure looks like: a count over repeated runs, with its denominator, its dates and the raw answers behind it.

The question is landing on agency desks now. In AgencyAnalytics' 2026 benchmark of 494 agency professionals, 66% reported client demand for AI search work and 41% reported problems measuring its return. The survey's sponsor sells an AI visibility tracker, so read it as a vendor's survey. This guide sets out what published studies found, what can be measured honestly, and the questions to put to any vendor, including us.

What the studies found

The same question rarely produces the same list

The most-cited test of consistency asked AI assistants the same recommendation questions thousands of times. Across 2,961 runs of 12 prompts, the identical list of brands came back in fewer than 1 in 100 runs, and the same list in the same order in fewer than 1 in 1,000, according to Search Engine Journal's report of the SparkToro study. SparkToro ran it with Gumshoe, which sells AI visibility tracking, and discloses the tie.

SparkToro's follow-up on whether AI visibility can be tracked at all draws two conclusions from the same work. A brand's position in any single answer is noise. How often a brand is named across many runs is far steadier, and it tightens as runs accumulate: on the figures reported, roughly 20 passes over 490 results put a rate within about one percentage point, while seeing the same two brands named together again in Google's AI Mode took around 124 runs.

The API is not the app

Most tools, and AnswerLift's automated collection, ask a model through its API. Three studies compared that with what the consumer apps show:

• An academic paper asked 80 questions 20 times each. Google results and the OpenAI API together recovered only about 26 to 28% of the URLs ChatGPT itself cited, and about half of its sites, and 44.5% of cited pages appeared in just 1 of the 20 runs (Search Engine Journal's summary).
• A UK banking study of 3,645 conversations found no search queries shared between the app and the API, only 12.3% of cited URLs used by both, and an app that leant much harder on comparison sites (32.2% of its citations against 11.0% for the API). Its authors score visibility from the app and treat the API as a diagnostic.
• Surfer, which sells collection from the consumer apps, compared 1,000 prompts across five products: brand overlap between app and API was 15.5 to 23.8%, and the app cited 12.1 sources per answer against 3.1. It is vendor-authored, and the vendor has a stake in the answer.

An API answer is therefore a real, repeatable observation of the model, not a picture of what a particular person sees in the app. Say which one you have. Our guide to the ChatGPT API versus the ChatGPT app goes through what each can and cannot tell a client.

Answers drift over days and weeks

Even one surface moves. Between observations, Google AI Overviews changed their wording 70% of the time and their cited sources 45.5% of the time, in Ahrefs research reported by Search Engine Journal in a piece Ahrefs sponsored. Sistrix measured weekly drift in cited domains of 56% in AI Mode and 74% in ChatGPT Search, with the UK in its sample. Profound, a vendor, puts monthly drift at 40 to 60%.

Variation is not only a matter of time. When Thinking Machines sent 1,000 identical requests at temperature zero, the setting meant to make output repeatable, it got 80 distinct outputs.

Scores and share of voice are contested

Ahrefs, which sells AI tracking, notes that most tools cap tracking at 50 to 100 prompts and advises treating AI tracking "less like rank tracking and more like polling". Stellar Search, a London agency, argues that a single visibility percentage is false precision. On the other side, LLM Pulse, also a vendor, argues that aggregated measures hold steady even when individual answers do not.

Both positions fit the evidence above. Aggregates over enough runs can be stable. Single answers, positions and percentages with no visible denominator are not.

What can be measured honestly

Put together, the evidence supports three kinds of figure and rules out two.

Supported:

• How often a brand is named across repeated runs of the same question. A rate means something once it rests on enough runs. With few runs it is a count, and should be shown as one.
• Which sources engines cite, and how often. Cited domains move week to week, so treat them as a distribution across runs rather than a fixed list. The sources that recur are the ones worth acting on.
• What a named person saw in a consumer app on a given day. It is one observation, not a trend, but it is the closest record of what a buyer sees.

Not supported:

• A position or rank taken from a single answer. The same order recurs so rarely that "third in ChatGPT" describes one draw, not a standing.
• A score or share of voice without its denominator. If a client cannot see how many runs, prompts and engines sit under a figure, nobody can check it.

The k-of-n method: counts with their denominators

The simplest honest format is "named in k of n runs": the brand appeared in k of the n answers the same engine gave to the same question, between two stated dates. It carries its own uncertainty. A count with its denominator and dates reads as what it is; a bare percentage invites the reader to supply a precision that was never measured.

This is how AnswerLift reports. A Visibility Sprint asks each question three times of each engine by default, through each provider's API reached via the OpenRouter gateway: the models behind ChatGPT and Claude with the vendor's own web search switched on, Gemini from model knowledge with no search grounding, Perplexity Sonar, which always searches, and five comparison models. Every answer is stored verbatim with its prompt, version, engine, model, search method, cited sources and timestamps. Findings read "named in k of n runs" with their dates, and no percentage is shown when n is under five.

Three runs is still a small sample, and we say so. It separates "never", "sometimes" and "every time"; it does not produce a rate you could quote to one decimal place.

Two further rules keep the counts honest. The consumer apps are never scraped: where a client needs to know what the ChatGPT app, Perplexity or Google AI Overviews showed, a person records it by hand and the record says so. Copilot, Grok, Meta AI and AI Overviews are not measured automatically at all. And a named reviewer signs off every record before it can reach a client: unverified records are excluded from the client report by the query that builds it, not hidden by the page. The full method, including what it cannot tell you, is on the methodology page.

Seven questions to ask any AI visibility vendor

A review of how AI visibility tools collect their data on metehan.ai argues that disclosure matters more than whether a tool uses the API or the app, and that one run a week is a poll rather than a measurement. Inspired by that review, this is the checklist we suggest agencies put to every vendor in writing, AnswerLift included:

1. API or app? Does the tool ask models through their APIs, drive the consumer apps, or both, and is every result labelled with which?
2. How many runs per prompt? How many times is each question asked of each engine, and how often is that repeated?
3. Which models, exactly? Which model and version answered, and was web search switched on?
4. Whose view? What location, language, persona or signed-in state is each question asked from?
5. What is n? Does every figure show the number of runs, prompts and engines behind it?
6. Can I read the raw answers? Can you open the verbatim answer and its cited sources behind any number?
7. Can I export it? Can you take the underlying records out in a form your team can check?

Some vendors already answer part of this in public: Peec, for example, documents that it runs prompts daily because answers vary. The answers will not make a tool right, but they tell you how much weight its numbers can bear. Larger clients are asking the same thing: in a survey by Conductor, itself a vendor, trust in visibility data was the top technology pain for the enterprise marketers it surveyed, as reported by MarTech Series.

How to word it in a client report

Clients ask for a number, and the evidence says you can give them one, as long as it comes with its working. Three habits make the difference:

• Lead with counts and dates. Write "named in [k] of [n] runs of the model behind ChatGPT, [first date] to [last date]" rather than a percentage.
• Label the surface. Say whether each record is an API answer or a person's observation of an app, and which model answered.
• Keep positions out of the headline. If position matters to the client, report the range of positions across runs next to how often the brand was named, never a single rank.

Our AI visibility report template sets out the sections, the caveat wording and what to leave out.

See it on one of your own clients

If you want to see this on a real brand before buying anything, the free AI Visibility Snapshot puts three buyer questions to the models behind ChatGPT, Claude, Gemini and Perplexity, and a person checks every answer before it reaches you, with the sources each engine cited. Each question is asked once, so treat it as a first look rather than a rate.