Why does ChatGPT give different answers to the same question, and how can you measure mentions?
ChatGPT gives different answers to the same question because it and other assistants can vary in how they generate answers, may run a fresh web search for each question and are updated over time, so the same question often gets a different list of businesses. You can still measure it: count how often your business is named across many answers, read that share alongside its likely range, and compare it over time instead of trusting one answer.
In short
- Answers vary because text is generated with some randomness, web searches can return different pages, systems are updated, and location, wording and personal history can matter.
- In our data, the same question asked again in a later check did not return an identical list in any of 42 pairs, but the named or not named verdict stayed the same in 38.
- A single answer is a sample. Count the share of answers that name you and read it with its likely range.
- Being named in 3 of 24 answers gives a 90% likely range of about 5 to 27%; being named in 15 of 120 gives a range of about 8 to 18%.
- Treat a change as evidence of a shift only when it exceeds normal variation: 3 of 24 rising to 6 of 24 does not pass that test, 3 of 24 rising to 8 of 24 does.
On this page
- Why ChatGPT gives a different answer to the same question
- What we saw when we asked again
- Count across many answers instead of trusting one
- The likely range: how precise is a share?
- Why 24 answers give a wide range, and how more answers narrow it
- Comparing two checks: real change or normal variation?
- How to measure despite the variation
ChatGPT can give different answers to the same question mainly because it writes every answer fresh, with some randomness in how it chooses words, and because it may search the web again for each question and find different pages. Updates to the system, your location, the exact wording and, if you are logged in, your history can change the answer too. That does not make measurement impossible: it means counting across many answers instead of relying on one.
This guide explains the causes in plain words, shows what we saw when the same questions were asked again, and walks through the two tools that make a changing answer measurable: a likely range and a test for real change.
Why ChatGPT gives a different answer to the same question
The causes below are general, widely described behaviour of AI assistants. The details differ between providers and change over time, and providers do not publish all of them, so treat this as an overview of what can play a part, not as a precise account of any one system.
Some randomness in how text is written
Language models write an answer piece by piece, and at each step there are usually several plausible continuations. Consumer assistants may allow some chance in that choice, so the same question can produce different wording, a different order and different businesses. A bike shop in Bern can appear in one list and be missing from the next without the shop having changed anything.
A new web search, possibly
Assistants such as ChatGPT, Google AI Mode and Copilot can look things up on the web while they answer. Which results come back, and which of them the assistant reads, can differ between two requests. A different article in the mix can bring different businesses into the answer.
Updates to the system
The companies behind these assistants change their models and settings from time to time, often without announcing each change. An answer from last month may have come from a different version of the assistant, so a change over several weeks is not necessarily caused by anything a business did.
Personal settings and history
Logged-in users may get answers shaped by memory, earlier chats or custom instructions, where those features are switched on. Two customers asking the same question may therefore see different answers. CheckAI asks questions while logged out, which removes this factor from the measurement but also means a logged-in customer may see something else.
Location, language and wording
Where a question seems to come from, and the language it is asked in, can change which businesses look relevant. In our setup only Google AI Mode accepts a country setting. ChatGPT, Gemini and Copilot get no location from us, so the place is written into each question. Wording matters as well: "tax adviser in Munich for freelancers" and "who can help me with my tax return in Munich" are different questions to an assistant and can get different lists.
What we saw when we asked again
Our data Two businesses were checked a second time with the same questions, in a later check on the same day. That gave 42 pairs of answers with the same business, the same question and the same assistant. The list of named businesses was identical in none of the 42 pairs. On average the two lists overlapped by 24%: of all the businesses named in either answer, about a quarter were named in both.
Our data The named-or-not verdict for the business being checked changed less often. Whether it was named or not stayed the same in 38 of the 42 pairs (90%), mostly because it was not named either time, and it flipped in 4. Different assistants agreed even less than repeats of the same one: for the same question, any two assistants shared on average about 11% of their names, and together the four named a median of 17 different businesses per question.
Count across many answers instead of trusting one
If one answer can change from one hour to the next, a single answer is a sample, not a verdict. A more useful number is the share of answers that named your business. That is why a CheckAI report leads with a count such as "named in 3 of 24 answers", split by assistant, rather than with one answer.
You can see such counts for single industries, for example in the tests for bike shops and accountants. The same logic applies if you check by hand. Asking one question once and reading one list tells you about that moment. Asking the same set of questions several times and counting tells you about the pattern.
The likely range: how precise is a share?
A share from a small number of answers is uncertain, so our reports show it with a likely range (see the AI visibility glossary): a 90% Wilson interval. In plain words, it is the span of true rates that could reasonably have produced what was counted. If answers are independent and comparable, ranges built this way contain the true rate roughly 9 times out of 10 across repeated checks.
Worked example. A business is named in 3 of 24 answers, which is 12.5%. The 90% likely range is about 5 to 27% (5.1 to 27.5% before rounding). The 12.5% is an estimate, not an exact rate; the likely range runs from about one in twenty to a little over one in four answers. Even 0 of 24 still leaves a range of 0 to about 10%, so a check with no mention does not prove that no customer ever sees the business named.
The range treats every answer as a separate draw. Answers to the same questions are related, so treat the ranges, and the change test below, as rough guides rather than exact cut-offs.
Why 24 answers give a wide range, and how more answers narrow it
The width of the range depends mostly on how many answers there are. The table keeps the rate the same, 12.5%, and changes only the number of answers.
| Named in | Share | 90% likely range | Width |
|---|---|---|---|
| 3 of 24 answers (6 questions, 4 assistants) | 12.5% | 5 to 27% | 22 points |
| 6 of 48 answers | 12.5% | 7 to 22% | about 16 points |
| 15 of 120 answers (30 questions, 4 assistants) | 12.5% | 8 to 18% | 10 points |
| 30 of 240 answers | 12.5% | 9 to 16% | 7 points |
Five times as many answers (24 to 120) cut the width by more than half. Doubling again to 240 helps less. That is the practical case for asking more questions per check and repeating checks: larger samples generally narrow the range, with the biggest gains coming early.
Comparing two checks: real change or normal variation?
When a share moves between two checks, the question is whether the move is bigger than the variation you would expect anyway. CheckAI answers it with a two-proportion test at the 90% level: it compares the two shares and asks how likely a gap of that size would be if the true rate had not changed at all. If the gap is too large to be explained by chance at that level, the change is flagged for review; otherwise the test finds no clear evidence of a change.
Four examples, all starting from a rate of 12.5% and computed with the same test:
| First check | Second check | Result at 90% |
|---|---|---|
| 3 of 24 (12.5%) | 6 of 24 (25%) | No clear evidence of change |
| 3 of 24 (12.5%) | 8 of 24 (33%) | Statistically significant change |
| 15 of 120 (12.5%) | 24 of 120 (20%) | No clear evidence of change |
| 15 of 120 (12.5%) | 27 of 120 (22.5%) | Statistically significant change |
With 24 answers, a rise from 3 to 6 mentions does not give clear evidence of a change; a rise to 8 passes this test. With 120 answers, a smaller step, from 12.5% to 22.5%, already passes. A test at the 90% level can still flag a change when nothing has changed: for one comparison under its assumptions, this happens about 1 time in 10. Related answers can affect that rate, and making many comparisons increases the chance of at least one false alarm. A single statistically significant change is a reason to look closer. A repeated pattern across checks can provide stronger evidence.
How to measure despite the variation
Advice These habits make the numbers comparable from one check to the next, whether you use a tool or a notebook.
- Keep the questions the same. Change the wording and you are measuring a different question. If you add questions, compare old and new separately. The guide on how to check if ChatGPT recommends your business shows how to write them.
- Keep the settings the same. Same assistants, same language, same place in the question, logged out every time.
- Count shares, not single answers. Note named or not for every answer and work with "k of n", not with the one answer you happened to read.
- Track over weeks. A fixed rhythm, for example weekly or every two weeks, shows a trend that one check cannot.
- Do not react to one answer. A bakery in Zürich that drops out of one ChatGPT answer has learned very little. Look at the share, its range, and whether the change passes the test.