Intuitively, we all understand the importance of sample size. You wouldn’t do a presidential poll by only calling 5 voters. You need to call 5,000 people to get to statistical significance in a national poll. It turns out the same thing is true in AI. Every time you ask a question to a large language model like ChatGPT or Claude or Gemini, the model gives a slightly different answer. This happens because the models are probabilistic - they vary their responses on every reply (you can read more on that here).
This is hugely important for brands thinking about how they show up in AI search results. Asking a question once, or 10 times, isn’t enough. Here’s why.
Let’s imagine a consumer who is asking ChatGPT, “What are the best portable speaker brands?” Here’s what that search looks like if you do it once. Five brands are recommended. Good enough, right?
But the data changes dramatically if you ask the same question 10 times. In this data, you can see that four new brands are introduced in the results. Sonos, Bang & Olufson, Marshall and Apple are added to the list of recommended products. Turns out, Sony isn’t ChatGPT’s favorite brand - Bose, JBL and Ultimate Ears are.
The same changes occur when you increase the sample size up to 50 times, or to 100. New brands are added to the list. Harmon Kardon, Klipsch, Tribe and AOMAIS are each mentioned once in those 100 queries. In this category, after 100 searches, there are diminishing returns in terms of new brand mentions.
Why does this matter? Imagine you’re the marketing team at Bang & Olufsen. With low sample sizes, it seems your brand has a huge awareness problem with AI. But it doesn’t - at 100 samples, Bang & Olufsen is being recommended 38% of the time.
Many marketers today are thinking about AI search the same way they’ve thought about traditional search on Google. While there are some similarities, the need for sampling is a massive difference. To monitor your brand the right way in AI, you need to sample at scale - a handful of queries on every keyword won’t produce meaningful results.
