How much variation between repeated runs would you consider acceptable?
If you run the same prompt multiple times and the AI gives different answers, how much variation would you consider acceptable before you stop trusting the measurement?
For example:
10 runs → brand mentioned 7 times
Would you consider that a reliable 70% visibility signal?
Or would you want:
20+ runs?
results across multiple days?
different locations?
a confidence/range rather than one percentage?
And importantly, what would you consider a meaningful change?
If visibility moves from 60% → 65%, is that something you'd act on, or would you need a much larger change before believing it wasn't just model variance?
Curious how people actually handle this today.