ChatGPT API vs the ChatGPT app: which should you measure?

A client asks a simple question: when a buyer asks ChatGPT for a recommendation, does our brand come up? Every tool that answers it has to choose how to ask. It can send the question to the model behind ChatGPT through OpenAI's API, it can drive the ChatGPT app with browser automation, or a person can open the app and look. These are three different measurements, and the published evidence says they do not agree closely enough to be treated as one.

This guide sets out why the API and the app differ, what three studies found, what each method can honestly tell a client, and a protocol that uses both without blurring them. AnswerLift sells an API-led service with hand-recorded checks alongside it, so read the comparison with that in mind; the limits of our own approach are stated near the end.

Why the app and the API give different answers

The model behind ChatGPT is only part of what a person sees in the app. The consumer product adds its own system prompt, memory of earlier conversations, personalisation and model routing, and it decides for itself when to search the web and what to search for. An API call made by a monitoring tool has none of that history and runs whatever search configuration the tool chose.

Search is where the gap is widest. When an assistant searches, it writes its own queries, often several at once (a "fan-out"), and cites what comes back. In one UK banking study, the app and the API never issued the same search query across 3,645 conversations.

Even the API on its own does not repeat itself. Thinking Machines Lab sent 1,000 identical requests at temperature 0 and got 80 distinct outputs. Answers also move over time: SISTRIX reports weekly drift in cited domains of 56% in Google's AI Mode and 74% in ChatGPT Search, with UK data included.

So any single answer, from either route, is one sample of a moving system. The useful question is what each route samples.

What three studies found

An academic comparison of API and app citations

A 2026 paper on arXiv asked 80 questions 20 times each and compared what ChatGPT cited with what could be recovered from Google results plus OpenAI's API. Together they recovered only about 26–28% of the URLs ChatGPT cited, and about half of its sites. The same study found that 44.5% of pages appeared in just one of the 20 runs. Search Engine Journal's write-up summarises the result as the API only partly matching ChatGPT's citations.

A UK banking study

genezio's study of 3,645 conversations about UK banking found no shared query fan-outs between the app and the API, and only 12.3% of cited URLs were used by both. The app leaned far more on comparison sites: 32.2% of its citations, against 11.0% through the API. The authors score visibility from the app and use the API as a diagnostic.

A vendor study

Surfer, which sells tracking based on scraped AI answers, compared 1,000 prompts across five AI products. Brand overlap between scraped app answers and API answers was 15.5–23.8%, and the apps cited 12.1 sources against 3.1 through the API. Surfer has a commercial interest in this result, so weigh it accordingly; its direction matches the two studies above.

What they agree on

All three point the same way. If the client's question is "what does the ChatGPT app cite when a buyer asks this?", API answers are a weak proxy: they miss most of the app's sources and favour different kinds of site. That is a real limitation, and a tool that reports API answers as "what ChatGPT says" is overstating what it measured.

They do not show that API answers are useless. An API answer is a repeatable sample of what the underlying model says when asked in a controlled way, with the model and the search setting known. That makes it the right instrument for some questions and the wrong one for others.

What each method can and cannot answer

Rates over repeated runs are the defensible unit for the first two columns. SparkToro's study, co-run with the tool vendor Gumshoe, found the identical list of brands came back in under 1 in 100 runs; SparkToro's follow-up concludes that a single position is noise and that a rate over many runs is steadier. The reasoning is set out in our guide to why AI answers change every time.

Ask how a tool collects before you trust its numbers

Most tools do not say which method they use. Of the vendor pages we read on 1 November 2026, four said how they collect answers: Peec AI automates the consumer interfaces, Promptwatch and SE Ranking capture the interfaces, and Conductor describes an API-first approach. Peec's documentation also explains that it collects daily because answers vary. Our comparison of AI visibility tools for agencies lists what each vendor publishes.

One analysis of how AI visibility tools collect data argues that disclosure matters more than the choice between API and interface. Before you put a tool's figure in front of a client, get written answers to these questions:

• Is each answer collected through an API, by automating the app, or by a person?
• Which model and which search setting produced it, and is that recorded on every answer?
• How many runs per prompt sit behind each figure, and over which dates?
• For interface capture: signed in or not, which location, and how are sessions created?
• Can you see the raw answer and its cited sources, not just a score?

If a vendor cannot answer these, the number on its dashboard cannot be defended in a client meeting.

A hybrid protocol you can defend

1. Write unbranded buyer questions. Ask what a buyer would ask before they know the brand, in the client's own market.
2. Run them repeatedly through the API. Ask each question several times of each engine, and report "named in k of n runs" with the dates and the model. Do not turn small counts into percentages, and do not report a position from a single answer.
3. Hand-check the questions that matter most. For the few prompts the client cares about most, have a person ask them in the ChatGPT app, Perplexity and Google AI Overviews, and record the answer with the date, location and whether they were signed in.
4. Keep the two kinds of record apart. Show API rates and app observations side by side, each labelled. Never merge them into one number.
5. Re-run on a schedule. Because answers drift, every figure carries a date. Compare like with like over time, and treat small movements as possible noise.
6. Say what you did not measure. Name the assistants and surfaces that were not checked, and why.

The protocol does not prove that a change on a client's site caused a change in an answer. It records what was observed, how, and when, which is what a client report can honestly carry.

How AnswerLift applies this, and what it does not measure

A Visibility Sprint asks the client's buyer questions of the models behind ChatGPT, Claude, Gemini and Perplexity through the OpenRouter gateway, plus five comparison models. The model behind ChatGPT is asked with OpenAI's own web search tool switched on, and Claude with Anthropic's; Perplexity's Sonar always searches. Gemini answers from model knowledge with no search grounding, because Google Search grounding is not available through that route. Each question is asked three times of each engine by default, and results are shown as "named in k of n runs", with no percentage until a figure rests on at least five runs.

Every API record is labelled as an API answer, with the exact model, the search method, any sources its search cited and the time. AnswerLift never scrapes or drives the consumer apps. Where a client wants to see exactly what a person sees, a person records what they saw in the ChatGPT app, Perplexity or Google AI Overviews as a separate, labelled record. Copilot, Grok, Meta AI and Google AI Overviews are not measured automatically.

Every record, collected or hand-recorded, is signed off by a named reviewer before it can appear in a client report. The full method is on the methodology page, the Sprint's scope and price are on the pricing page, and you can see the approach on one of your own clients with a free AI Visibility Snapshot.

The short answer

For ChatGPT brand monitoring, measure both and label both. Use repeated API runs to say how often the model behind ChatGPT names a brand, with the denominator and the dates on every figure. Use hand-recorded app checks to show a client what a person actually sees on the questions that matter most. Do not present either as the other.