How to read your A/B test results
The Algolia A/B testing feature helps you set up an A/B test to assess whether your search strategy is successful. Reading the results and trends from your test might seem straightforward, but there are several things to bear in mind when interpreting results.
Understanding how A/B testing works
How results are computed
A/B test results are computed from click-through and conversion rates based on the events you send to the Insights API. This computation can lead to different results from what you have on your business data.
Automatic outlier exclusion
When A/B testing, outlier traffic can significantly skew the results, making it unrepresentative of real user traffic. Algolia’s A/B testing feature automatically excludes outlier users when calculating A/B test result metrics.
A user qualifies as an outlier if the number of tracked searches performed by that user is 7 standard deviations (σ) more than the mean (μ) number of searches per user for the entire A/B test. A tracked search is a search with the
clickAnalytics parameter set to
To check the number of outliers and the number of tracked searches that were removed from each variant, hover over the Tracked Searches and Tracked Users counts for each variant.
For example, when a bot scrapes your site, it can trigger search requests without clicking or converting. If this bot were assigned to the B variant for an A/B test, it would artificially increase the number of tracked searches. This increase would cause a larger-than-intended decrease in click and conversion rates for the test, even when an increase would be seen for real users. With outlier removal, these significant, undesirable sources of traffic will be excluded. This exclusion ensures that results will more accurately represent real-user searches and the impact of any changes in the variant.
Looking at your business data
As Algolia A/B testing is specific to search, the focus is on search data. It’s best to look at the A/B test results in light of your business data to assess your test’s impact. For example, you can cross-reference search and revenue data to compute your interpretation of conversion or to look at custom metrics.
When to interpret what
The A/B testing interface in the Algolia dashboard displays data starting from the second day of your test. Even if some numbers are showing, it doesn’t mean it’s time for you to draw conclusions. Here’s what you should look at before interpreting the A/B test results.
Is the data significant?
Before you interpret results and draw conclusions on what steps to take next, ensure that the numbers are significant. The significance score helps you assess whether the results are the product of chance or an accurate representation of how your users behave.
When the significance is higher than 95%, you can be confident that the results will happen in the future. You shouldn’t draw any conclusion if the significance is below 95%.
Is there enough data?
The larger your sample size, the better. You should run A/B tests for at least two business cycles and wait for the end of the test to draw conclusions. Experience proves the best results happen when the traffic represented by both variants is over 100,000 searches. If your overall A/B test traffic is below this, you might want to take its results with a grain of salt.
Is the split off?
When setting your A/B test, you assign a percentage of search traffic to each variant (by default, 50/50). The expected traffic split should be reflected in the displayed user count. For most A/B test configurations, the search count for each variant should match the traffic split. If there’s a noticeable discrepancy, there’s probably an issue.
For example, you could have a 50/50 split, ending up with 800,000 searches on one side and 120,000 on the other. The results are unreliable if you see a difference higher than 20% from the expected number. In such scenarios, you should investigate your A/B test setup to understand what’s happening. It could be due to outliers like internal IP addresses or external bots skewing the results, or a backend implementation not sending all the data.
If you believe something’s wrong with your A/B test results, there are some checks to identify what could be the root cause.
Analytics and events implementation
Since A/B tests rely on Click and Conversion events, ensure you’ve properly implemented them.
For example, you might want to check the following:
- Are you catching both click-through and conversion rates?
- Are there enough events on popular searches?
- Are there any errors in the Insights API Logs within the Monitoring section of the Algolia dashboard?
Sales or holidays (such as Black Friday) can affect your test, with more out-of-stock items, for example. If you see unexpected results, check whether you’ve been conducting the A/B test during a special period.
A/B test in a Dynamic Re-Ranking context
If you have launched an A/B test to evaluate Dynamic Re-Ranking’s effect, there are a few additional considerations:
- Make sure to launch the A/B test through the Dynamic Re-Ranking interface. If it’s not the case, Dynamic Re-Ranking must be opted-in for the index being tested.
- If using a replica, ensure re-ranking does have an impact: you should have queries re-ranked for this index. If not, change its events source index to the primary index for Dynamic Re-Ranking.
- Check whether Personalization is also enabled for the index you’re testing. The analysis isn’t optimal when Personalization is enabled: it counts traffic towards Dynamic Re-Ranking when it probably doesn’t matter for a given percentage of the traffic. Indeed, as soon as a Personalization user profile is detected, Dynamic Re-Ranking no longer has any effect.
- Dynamic Re-Ranking isn’t optimized for particular use cases like marketplaces with short-lived items, for instance. If this is your use case, you may see A/B test results that aren’t satisfactory.
Are you using distinct? If so, items are regrouped so Dynamic Re-Ranking has a lesser impact.
The data is no longer significant
The significance computation involves ratios, and there’s no guarantee that significance will be achieved even if the test is setup perfectly.
Your data could be significant at some point, and no longer significant some days later.
Some of the reasons for this are:
- Variability in the Data: A/B test results can be quite volatile, especially when the sample size is small. It’s possible that during the early stages of the test, the observed difference was due to chance, and as more data was collected, the true difference (if any) became clearer.
- External Factors: External events could influence the behavior of users. For example, if you’re testing changes on an ecommerce site, a major holiday, sale, or event could skew results temporarily.
- Sampling Bias: If there’s any change in the type or behavior of users during the test (for example, a change in traffic source, a product release, or a marketing campaign), it can influence results.
Some things to check in case the A/B doesn’t reach enough significance:
- Review Test Setup: Ensure that there have been no technical issues or changes during the test that could have affected the results. Make sure the groups are still being split correctly, and no outside influences have been introduced.
- Consider External Factors: Were there any external events or factors that could have influenced the results? Understanding these can help in interpreting the drop in significance.
- Increase Sample Size: If the sample size is small, continue running the test to collect more data. A larger sample size (for example, by running the test for a longer period of time) can give more accurate and stable results.
Click-through rate is going up and conversion rate is going down
This probably means that top search results don’t convert. Potential causes include out-of-stock products, items with the wrong picture, misleading descriptions, or unavailable sizes. In such cases, you can use filtering to eliminate unavailable products, fix the relevance implementation to prioritize certain items or clean up your data.
If possible, you should also look at your own business intelligence metrics to confirm/deny the results you’re seeing. If you’re looking at revenue, for instance, what’s the impact of the A/B test?
Click-through and conversion rates are both going down
You might reach the point where you have been through all the possible checks to see if there’s something wrong with your A/B test and Insights API implementation, and nothing seems to explain the results. Perhaps the settings you’re testing aren’t a good strategy for your search: keep iterating to find a successful strategy.
Contact the Algolia support team for further help.