> ## Documentation Index
> Fetch the complete documentation index at: https://algolia.com/llms.txt
> Use this file to discover all available pages before exploring further.

# How traffic splits affect A/B test results

> Learn how A/B test calculations account for even and uneven traffic splits.

When you create an A/B test, you choose how to divide traffic between variants. You can use an even split, such as 50/50, or send less traffic to an experimental variant, such as 90/10.

An uneven split doesn't bias the results. Algolia calculates each variant's metrics and statistical uncertainty independently from the traffic that variant receives. However, the smaller sample produces less precise estimates, so an uneven split usually takes longer to reach statistical significance.

<Note>
  **Key takeaway:** A 90/10 traffic split doesn't bias your results. It makes them less precise. Because the smaller variant collects data more slowly, the test usually needs more time to reach statistical significance.
</Note>

## Compare a 90/10 split with a 50/50 split

Suppose an A/B test receives 100,000 tracked searches. With a 90/10 split, the control receives 90,000 searches and the test variant receives 10,000.

Compared with a 50/50 split, the 90/10 test needs about 2.78 times as much total traffic to have the same chance of detecting the same effect. A test that needs 100,000 tracked searches with a 50/50 split would need about 278,000 with a 90/10 split, assuming similar rates and user behavior.

The smaller variant collects data more slowly, so the test takes longer to produce a statistically significant result. This difference affects how much data the test needs, not whether the comparison remains valid.

## Choose a traffic split

* Use a 50/50 split when you want to reach a given level of precision with the least total traffic.
* Use an uneven split, such as 90/10, if limiting exposure to an experimental change is more important than reaching results sooner.
* Expect the test to run longer as the traffic split becomes more uneven.

## How Algolia accounts for traffic splits

Algolia accounts for each variant's sample size when calculating statistical significance and confidence intervals. The calculations don't assume equal traffic across variants, so you don't need to correct the results of an uneven split.

For rate metrics, such as click-through rate and conversion rate, the smaller variant contributes more statistical uncertainty. For continuous metrics, such as revenue per search and average order value, the calculation accounts for both the sample size and the variability within each variant.

The Algolia dashboard shows a 95% confidence interval for the relative change between each test variant and the control. A smaller sample produces a wider confidence interval because its estimate is less precise. The control is normalized to zero, and a test variant is statistically significant when its confidence interval doesn't overlap zero.

For metrics with significance testing, Algolia compares each test variant with the control and can detect whether the test variant performs better or worse.

## Statistical calculation details

Expand these sections if you need the formulas and implementation details behind the results.

<AccordionGroup>
  <Accordion title="Why a 90/10 split needs more traffic">
    If the underlying rates are similar, the test variant's variance term is about nine times the control's variance term. Most of the statistical uncertainty comes from the smaller variant.

    Compared with a 50/50 split with the same total traffic:

    * The variance is about 2.78 times larger.
    * The standard error is about 1.67 times larger.
    * The test needs about 2.78 times as much total traffic to reach the same statistical power for the same effect size.
  </Accordion>

  <Accordion title="Significance testing">
    Algolia uses a two-sided hypothesis test, which detects whether the test variant performs better or worse without assuming a direction in advance.

    After calculating the test statistic, Algolia derives the two-sided p-value from the standard normal distribution:

    $$
    p = 2 \left(1 - \Phi\left(\lvert z \rvert\right)\right)
    $$

    With one control and one test variant, a result is statistically significant when its p-value is 0.05 or less. Tests with more than one test variant use a corrected threshold. For more information, see [Multi-variant testing](/doc/guides/ab-testing/what-is-ab-testing/in-depth/multi-variant-testing).
  </Accordion>

  <Accordion title="Rate metric calculations">
    Rate metrics include click-through rate, conversion rate, add-to-cart rate, and purchase rate. For these metrics, Algolia uses a two-proportion z-test with an unpooled standard error.

    The observed rate for each variant is:

    $$
    \hat{p} = \frac{\text{number of events}}{n}
    $$

    The sample size, $n$, is the number of observations for that variant. Depending on the metric, an observation is a tracked search or a user.

    The test statistic compares the rates for the control variant, $A$, and the test variant, $B$:

    $$
    z = \frac{\hat{p}_B - \hat{p}_A}{SE}
    $$

    $$
    SE = \sqrt{\frac{\hat{p}_A(1 - \hat{p}_A)}{n_A} + \frac{\hat{p}_B(1 - \hat{p}_B)}{n_B}}
    $$

    Each variant contributes to the standard error based on its sample size. A smaller sample contributes more to the standard error, so the test requires more data or a larger observed difference before the result becomes statistically significant.
  </Accordion>

  <Accordion title="Revenue and other continuous metric calculations">
    Continuous metrics can take values other than zero or one. Examples include revenue per search and average order value.

    Algolia summarizes each variant with its sample mean, standard deviation, and sample size. It then uses a large-sample, unequal-variance z-test:

    $$
    z = \frac{\bar{x}_B - \bar{x}_A}{SE}
    $$

    $$
    SE = \sqrt{\frac{s_A^2}{n_A} + \frac{s_B^2}{n_B}}
    $$

    In these formulas, $\bar{x}$ is the sample mean, $s$ is the sample standard deviation, and $n$ is the sample size. The sample size is the number of tracked searches for revenue and revenue per search, and the number of transactions for average order value.

    The standard error has the same form as the one used by Welch's t-test. Algolia uses the standard normal distribution, rather than Student's t distribution, to calculate the p-value.

    Revenue calculations use winsorized values to reduce the effect of unusually large purchases. For more information, see [Revenue winsorizing](/doc/guides/ab-testing/how-to-read-your-a-b-test-results#revenue-winsorizing).
  </Accordion>

  <Accordion title="Confidence interval calculations">
    Algolia calculates 95% confidence intervals for the relative change between each test variant and the control. The calculation depends on the metric type:

    * For rate metrics, such as click-through rate and conversion rate, the interval uses the relative ratio between the test variant and the control. The calculation uses a log-transformed ratio and the delta method.
    * For continuous metrics, such as revenue and average order value, the interval uses the relative difference between the test variant and the control. The calculation uses the delta method to estimate the standard error.

    Both calculations use each variant's sample size independently. A smaller sample produces a wider confidence interval because its estimate is less precise.
  </Accordion>
</AccordionGroup>

## Check your traffic distribution

Algolia assigns users to variants randomly and uses their user tokens to keep them in the same variant for the duration of the test. For most configurations, the observed search counts should approximately match the traffic split you selected.

If the difference is large, check your implementation. For more information, see [How to read your A/B test results](/doc/guides/ab-testing/how-to-read-your-a-b-test-results#is-the-split-off).