A/B testing requires careful interpretation.
Reliable tests help you determine whether a change has a meaningful effect.
A/B testing is grounded in statistical inference.
Algolia uses established, reproducible statistical methods.
These methods determine whether differences between the control and variants are likely due to the change or to chance.
The method and math at a glance
- Randomness: the assignment of any one user to scenario A or B is purely random.
- Confidence measures the certainty of the outcomes of a test.
Confidence is quantified using the p-value. For classic two-variant tests, a result is confident (statistically significant) when the p-value is below 0.05 (95 % confidence).
If you compare more than two variants, the p-value is stricter so the test results stay reliable.
To learn more, see Multi-variant testing.
- Mathematical formula: frequentist comparisons use a two-tailed test appropriate to the metric. Two-tailed tests detect differences in either direction: better or worse, without assuming which variant performs better.
- Relevance improvement: in practice you add variants because you hypothesize one of them performs better than the control. Multi-variant tests let you try several ideas at once and show which, if any, delivers better impact.
For details about how these calculations handle even and uneven traffic distributions, see How traffic splits affect A/B test results.
Measurement units and statistical tests
Search outcome rates count searches, while user outcome rates count users.
An average of individual users’ rates is a different measure from the share of users with an outcome.
For definitions and examples, see Per-search and per-user metrics.
Frequentist inference depends on the metric:
The dashboard doesn’t display every metric listed here.
The statistical method configured for the test determines how to interpret its results.
Frequentist p-values and confidence intervals describe different quantities from Bayesian probabilities and credible intervals.
For Bayesian results and the primary metric’s evidence status, see Bayesian experimentation.
Statistical significance or chance
When you run your tests, you may get results that show a 4% increase in one of the measured metrics.
Statistical significance is concerned with whether the 4% increase is chance or real.
The statistical concern is whether your sample group truly represents the larger population.
Does the 4% make sense only for that sample group or does it reasonably predict the behavior of the larger population?
If the sample doesn’t represent the larger population,
then your results are due to chance.
Statistical significance (the confidence indicator) distinguishes chance from a real change.
When a result reaches statistical significance,
the difference between the control and variants is less likely to be due to chance.
It provides evidence that the change may also affect the larger population.
Large, distributed samples
Large data samples are necessary to reach confidence.
When flipping a coin 1,000 times, you can expect a close to 50:50 ratio of heads and tails.
If you flip it only a few times, the ratio can be heavily skewed.
You could flip heads three times in a row, but that’s unlikely if you flip it 1,000 times.
A larger sample size stabilizes results and increases confidence.
For user outcome rates, each user contributes one yes-or-no observation per variant during the measurement period, regardless of how many searches they perform.
When to run an A/B test
Be careful when you test. Testing during a sales campaign, a major holiday, or some other exceptional event, can undermine the reliability of your results. Last modified on November 16, 2026