Large language models (LLMs) are probabilistic. Ask one the same question twice, and you will get two different answers, the result of the LLM probabilistically sampling from its own vocabulary, knowledge and search results.
That matters a lot if you are a brand trying to understand what LLMs are saying about you and your competitors. If you ask an LLM to recommend brands in your category and your brand shows up, that does not mean it always shows up. Conversely, if your brand does not show up, that does not mean it never shows up.
It is essential, therefore, to repeatedly sample the same prompt. Only then can you reliably detect if your brand usually shows up in LLM responses to that prompt. The more you repeat the prompt, the more certainty you have about how visible your brand is.
How many samples is enough?
Consider the following prompt, “I’m in the market for an SUV. Any recommendations?” How visible is Acura in results to this prompt?
If you ask ChatGPT this prompt five times, Acura shows up in about 10% of responses, but at only five repetitions, the margin of error is a massive 27 points, meaning the brand’s visibility score could be as high as 37% and as low as 0%. Ask 12 times, the margin of error drops to 12 points. Ask 100 times, it falls to about 6 points.
This is a statistical reality regardless of brand or product category. We asked ChatGPT about brands in grills, electric vehicles, refrigerators, mattresses and dishwashers, and in each category, the larger the sample size, the smaller the margin of error. In other words, the more times a prompt is repeated, the more certainty brands have about their visibility in AI search.
Obviously, you cannot repeat a prompt infinitely. You have to stop at some point. Consistently, we see tight margins of error at 100 repetitions and diminishing returns for repetitions beyond this point.
How do I get insights instead of noise?
For any given prompt, the differences from one LLM response to the next are largely random. If ChatGPT recommends Toyota in its first response to “I’m in the market for an SUV. Any recommendations?” but not the second, that’s due to ChatGPT’s probabilistic choices, not Toyota or its Generative Engine Optimization (GEO) strategy. Response-to-response differences are mostly noise.
By contrast, the differences between the response to one prompt and another are meaningful differences. If ChatGPT mentions Toyota when responding about reliable SUVs but not high-performance SUVs, that is something Toyota’s marketing team can take action on via GEO strategies or ads.
But unless you have a meaningful sample of responses per prompt, each unique prompt you ask introduces new noise to your data.
Why isn’t asking lots of different questions enough?
Some GEO platforms argue you don’t need to sample repeatedly as long as you run enough unique prompts. Running more unique prompts can help get a clearer picture of a brand’s overall visibility. After all, more prompts means a bigger sample size for the overall brand picture.
But more prompts without more iterations per prompt inherently means more noise, making it harder to find the topic and prompt terms that are a brand’s strengths and weaknesses.
If your prompt topics are a mile wide and an inch deep, meaning they span multiple categories but are only sampled once, you lose the ability to drill down into any of them. You have added more topics, but also more noise. Without repeated sampling of each topic, you can’t tell if the topic warrants a marketing focus.
Additionally, increasing the number of prompts increases the risk of adding misleading prompts, prompts that bias or muddy the results. Consider reliability. There are a handful of straightforward ways to ask models to list brands with the most reliable SUVs, but the more unique prompts you add, the more likely you are to include prompts that don’t aid your understanding of this topic.
For example, you could ask prompts like “Which SUV brands break down the most?” or “Is Toyota a reliable SUV brand?” or “What are the most reliable SUVs under $50,000?” With the first prompt, high visibility would actually be a negative (the model is naming brands that are not durable); with the second, Toyota will always be mentioned because it was named in the prompt; with the third, you’ve now introduced price into what was previously a question about durability.
While each of these prompts can aid in understanding an aspect of an SUV brand’s visibility, they inhibit understanding the original purpose of the prompts: identifying which brands LLMs recommend for reliable SUVs. Without careful prompt pruning and meaningful sampling, you will lose your insights amongst statistical noise.
Insights, the Data Science Way
Ask two people a question, get two different answers. Ask an LLM the same question twice, get two different answers. The only way to see past response-to-response noise is to repeat each prompt enough times to have a statistically meaningful sample of responses.
Crafting a tight list of carefully worded prompts protects the results from leading questions, mixed semantics and other prompting taboos. Repeating those prompts protects the insights from noise.
When reviewing GEO partners make sure to choose a platform that baked sampling into the core architecture and has considered methodology.
Methodology
At Evertune, we track thousands of brands across LLMs by running millions of prompts a day. For this analysis, we ran 10,700 prompts on ChatGPT.
Evertune is the AI marketing platform for brands that want to own the AI customer journey. Evertune analyzes prompt responses at scale across all major LLMs, ChatGPT, Claude, Gemini, AI Overviews and more, to deliver statistically significant visibility data, then closes the loop with tools to act on it: website optimization, data-driven content creation, most influential sources, and paid activation through affiliate and programmatic AI retargeting partners. Where most tools tell you where you stand, Evertune tells you what to do about it. Founded by early executives of The Trade Desk and backed by $20M from leading investors.
Don't forget to share this post
