Understanding how A/B testing works
How results are computed
Algolia computes A/B test results from searches and the click and conversion events you send to the Insights API. This computation can lead to results that differ from what you see in your business data. To learn more about these rates, see:Per-search and per-user metrics
A tracked search hasclickAnalytics set to true.
A tracked user has at least one tracked search that passes the test’s filters.
When empty-query exclusion is enabled, searches with empty queries don’t contribute to user metrics.
Search rates measure the percentage of tracked searches with at least one qualifying outcome.
User outcome rates measure the percentage of tracked users with the measured outcome in that variant during the measurement period.
Users without that outcome still count in the denominator if their tracked searches pass the test’s filters.
Repeated searches or outcomes don’t count the same user more than once for a user outcome rate.
Add-to-cart, purchase, and revenue metrics require access to revenue analytics.
For example, two users make a total of 11 tracked searches that pass the test’s filters.
- The first user makes 10 searches and converts on one.
- The second user makes one search and doesn’t convert.
- No-results rate per user: includes users who encountered both searches with results and searches without results. Lower is better.
- Click position per user: each clicking user’s average click position, averaged across users who clicked. Users without clicks aren’t included. Lower is better.
Automatic outlier exclusion

Revenue winsorizing

Revenue per user
Revenue per user is based on the total tracked purchase revenue for each user in the selected currency. By default, Algolia caps each user’s total at approximately the ninety-ninth percentile of positive user revenue totals. It calculates the percentile across all variants for the selected currency. For example, if a user has two purchases of $80 and the cap is $100, that user contributes $100. Revenue per purchaser uses the same capped user totals. Revenue per user includes tracked users who purchased in the selected currency and tracked users who made no tracked purchases in any currency. Users without tracked purchases contribute zero revenue. Algolia excludes users whose tracked purchases were only in other currencies from that currency’s population. Revenue per purchaser includes only the purchasers in that currency.Looking at your business data
As Algolia A/B testing is specific to search, the focus is on search data. It’s best to look at the A/B test results in light of your business data to assess your test’s impact. For example, you can cross-reference search and revenue data to compute your interpretation of conversion or to look at custom metrics.When to interpret what
Algolia A/B tests begin to show data in the dashboard after either 7 days have passed or 20% test completion, whichever happens first. This is because A/B test data can be unstable at the start. Comparing results before the test reaches 7 days or 20% completion could lead to wrong conclusions. Even if some data is visible, it doesn’t mean that it’s the right time to make decisions based on them. Consider the following factors before interpreting the results of an A/B test.Bayesian metrics
The probability to be best measures the likelihood that a variant outperforms the control for a given metric. In Algolia A/B tests, a probability of at least 95% provides strong evidence that the variant outperforms the control. The credible interval shows a range of plausible values for the true uplift of a metric. A 95% credible interval means there’s a 95% probability that the true uplift is within that range. To learn more about how these metrics are derived, see Bayesian experimentation.
Is there enough data?
As an experiment collects more searches, its estimates become more precise. Use the evidence status to assess whether the experiment has collected enough evidence to interpret its results. An evidence status appears on the overview page and at the top of each results page. How Algolia determines this status depends on the statistical method used for the experiment.- For Bayesian experiments, the status is based on e-values.
- For frequentist experiments, the status indicates whether the test meets the configured minimum detectable effect.
Confidence in experiment results
Before interpreting the results or deciding what to do next, check the experiment’s confidence value. For more information, see How A/B test scores are calculated. While an A/B test is running, its confidence value can change as new data arrives. Treat interim values as provisional. After the A/B test ends, the displayed confidence value no longer updates.Is the split off?
When setting your A/B test, you assign a percentage of search traffic to each variant (by default, 50/50). The displayed search count should reflect the expected traffic split. For most A/B test configurations, the search count for each variant should match the traffic split. If there’s a noticeable discrepancy, there’s probably an issue. For example, you could have a 50/50 split, ending up with 800,000 searches on one side and 120,000 on the other. The results are unreliable if you see a difference higher than 20% from the expected number. Algolia attempts to identify and ignore any unusual data (outliers) when calculating A/B test results. But if there’s a big gap between the traffic you expected and the traffic you got, investigate your A/B test setup. This helps you figure out why there’s a mismatch.How to interpret confidence intervals?
Each confidence interval is comparing the variant with control, and Algolia doesn’t show any variant to variant comparisons. Algolia implements a relative confidence interval, which normalizes the control to zero. Over time, the confidence interval decreases. A variant is statistically significant when the colored band doesn’t overlap zero.Troubleshooting
If you believe something’s wrong with your A/B test results, there are some checks to identify what could be the root cause.Analytics and events implementation
Since A/B tests rely on Click and Conversion events, ensure you’ve properly implemented them. For example, you might want to check the following:- Are you catching both click-through and conversion rates?
- Are there enough events on popular searches?
- Are there any errors in the Insights API Logs within the Monitoring section of the Algolia dashboard?
Seasonality
Sales or holidays (such as Black Friday) can affect your test, with more out-of-stock items, for example. If you see unexpected results, check whether you’ve been conducting the A/B test during a special period. For more information, see A/B test implementation checklist.A/B test in a Dynamic Re-Ranking context
If you’ve launched an A/B test to check Dynamic Re-Ranking’s effect, there are extra things to consider:- Make sure to launch the A/B test through the Dynamic Re-Ranking interface. Otherwise, you must opt in Dynamic Re-Ranking for the you’re testing.
- If using a replica, ensure re-ranking does have an impact: queries should be re-ranked for this index. If not, change its events source index to the primary index for Dynamic Re-Ranking.
- Check whether Personalization is also enabled for the index you’re testing. The analysis isn’t optimal when you enable Personalization: it counts traffic towards Dynamic Re-Ranking even when that traffic probably doesn’t matter. As soon as Algolia detects a Personalization , Dynamic Re-Ranking no longer has any effect.
- Dynamic Re-Ranking isn’t optimized for use cases like marketplaces with short-lived items. If this is your case, you may see A/B test results that aren’t ideal.
The data is inconclusive
The confidence calculation involves ratios. Even if you set up the test as intended, there’s no guarantee it reaches confidence. Your data could be trending confident one day but inconclusive some days later. Some of the reasons for this are:- Variability in the data: A/B test results can vary a lot when the sample size is small. At the start of the test, chance alone might explain the difference you see. As you collect more data, the true difference, if any, becomes clearer.
- External factors: outside events can influence how users behave. For example, if you’re testing changes on an ecommerce site, a major holiday, sale, or event could skew results for a while.
- Sampling bias: if the kind of user or their behavior changes during the test, for example due to a change in traffic source, a product release, or a marketing campaign, it can influence results.
- Review your test setup: make sure no technical issues or changes happened during the test that could affect the results. Confirm the groups still split as configured, and that no outside factors crept in.
- Consider external factors: were there any external events or factors that could have influenced the results? Understanding these can help you interpret confidence.
- Increase the sample size: if the sample size is small, keep running the test to collect more data. A larger sample size, for example by running the test for longer, can give more accurate and stable results.
- Adjust the split: if the sample size is large but confidence is still low, consider changing the split to increase the number of users in each group. For example, if the split is 70/30, try 60/40 or 50/50. Uneven splits can help limit exposure to a high-risk change, but they also need a larger sample size to reach confidence.