> ## Documentation Index
> Fetch the complete documentation index at: https://www.algolia.com/llms.txt
> Use this file to discover all available pages before exploring further.

# How A/B test scores are calculated

> A/B testing calculates reliable statistics using a standard, mathematical approach.

A/B testing requires careful interpretation.
Reliable tests help you determine whether a change has a meaningful effect.

A/B testing is grounded in statistical inference.
Algolia uses established, reproducible statistical methods.
These methods determine whether differences between the control and variants are likely due to the change or to chance.

## The method and math at a glance

* **Randomness:** the assignment of any one user to scenario A or B is purely random.
* **Confidence** measures the certainty of the outcomes of a test.
  Confidence is quantified using the [p-value](https://blog.minitab.com/en/adventures-in-statistics-2/understanding-hypothesis-tests-confidence-intervals-and-confidence-levels). For classic two-variant tests, a result is confident (statistically significant) when the p-value is below 0.05 (95 % confidence).
  If you compare more than two variants, the p-value is stricter so the test results stay reliable.
  To learn more, see [Multi-variant testing](/doc/guides/ab-testing/what-is-ab-testing/in-depth/multi-variant-testing).
* **Mathematical formula:** the same [two-tailed test](https://en.wikipedia.org/wiki/One-_and_two-tailed_tests) is applied to every comparison between the control variant and each test variant. Two-tailed tests detect differences in *either* direction: better or worse, without assuming which variant performs better.
* **Relevance improvement:** in practice you add variants because you hypothesize one of them performs better than the control. Multi-variant tests let you try several ideas at once and show which, if any, delivers better impact.

For details about how these calculations handle even and uneven traffic distributions, see [How traffic splits affect A/B test results](/doc/guides/ab-testing/what-is-ab-testing/in-depth/how-traffic-splits-affect-results).

## Statistical significance or chance

When you run your tests, you may get results that show a 4% increase in one of the measured metrics.
[Statistical significance](https://blog.analytics-toolkit.com/2017/statistical-significance-ab-testing-complete-guide/) is concerned with whether the 4% increase is chance or real.
The statistical concern is whether your **sample group** truly represents the **larger population**.
Does the 4% make sense only for that sample group or does it reasonably predict the behavior of the larger population?

If the sample doesn't represent the larger population,
then your results are due to chance.
**Statistical significance (the confidence indicator) distinguishes chance from a real change.**
When a result reaches statistical significance,
the difference between the control and variants is less likely to be due to chance.
It provides evidence that the change may also affect the larger population.

## Large, distributed samples

Large data samples are necessary to reach confidence.
When flipping a coin 1,000 times, you can expect a close to 50:50 ratio of heads and tails.
If you flip it only a few times, the ratio can be heavily skewed.
You could flip heads three times in a row, but that's unlikely if you flip it 1,000 times.
A larger sample size stabilizes results and increases confidence.
Each new search event clarifies the pattern and leads to a more reliable outcome.

## Sample diversity

**Be careful when you test.** Testing during a sales campaign, a major holiday, or some other exceptional event, can undermine the reliability of your results.