Create and launch a test
1
Choose a test type
On the A/B Testing page, click Create test.
Select what you want to compare:
- A feature: test whether a supported feature improves your results, or compare configurations of a supported feature. Available features depend on your app and permissions.
- Index configurations: compare search-time configurations for a single index, such as typo tolerance, synonyms, facets, geo search, and distinct.
-
Index vs. Index test: compare separate indices or replicas.
Use this option to compare settings that require separate indices, such as searchable attributes or custom ranking.
2
Build the test
Select the index that represents the current production experience.
This becomes the control for the test.For feature and index configuration tests, the control and variants share a baseline index.
Configure each variant with the change you want to measure.
For an Index vs. Index test, select a separate index for each variant.The dashboard checks every selected index:
- An index that’s already part of an active A/B test blocks the next step.
- An index without recent search traffic or events shows a warning. You can continue, but the test might not collect enough data for useful results.
When testing with replica indices,
use the main index, which receives search traffic, as the control.
Use the replica as a test variant.
A replica doesn’t receive traffic when used as the control because Algolia assigns variants during the .
3
Configure the test
Choose how much traffic to send to each variant.
Start with an equal split unless you want to limit exposure to an untested configuration.
The traffic percentages must total 100%.For supported test types, select a target metric and the smallest relative change you want to detect.
The dashboard uses your historical traffic and events to estimate the test duration.
If duration estimation isn’t available for the selected test type, set the duration manually.
For more information, see Understand sample size and test duration.
4
Review and launch the test
Enter a descriptive test name and an optional hypothesis.
Review the test type, variants, traffic allocation, target metric, and duration.
To change a section, click its edit button.If everything looks correct, click Launch test.
Understand sample size and test duration
Accurate sample size determination is essential for effective A/B testing and reliable results. The sample size estimation tool simplifies this process by estimating the number of searches needed to confidently detect meaningful differences between experiment variations.Sample size estimator
The sample size estimator uses your historical data to establish a baseline for performance metrics, such as conversion or click-through rates. Algolia uses these metrics to calculate the optimal sample size required for statistically significant changes. With fixed statistical parameters—80% power and a 5% significance level (adjusted in multi-variant testing), the tool ensures robust and reliable results. Based on your traffic patterns, Algolia recommends a test duration that corresponds to the required sample size, rounding up to the nearest week to account for variability in business cycles. Key features:- Historical baseline rate. The tool uses your existing data to determine the current performance metric, such as conversion or click-through rate, against the measured changes.
- Fixed statistical parameters. A statistical power of 80% and a significance level of 5% (adjusted for multi-variant testing) provide a balance between detecting real effects and minimizing false positives.
- Customizable metrics and effect size. Select the metric you want to measure, such as conversions or clicks, and define the minimum detectable effect size that aligns with your objectives.
Choose an appropriate effect size
The effect size is the smallest relative change in a metric you consider significant enough to act on. Choosing an appropriate effect size is essential for accurate and efficient A/B tests. For example, if you would adopt a change only if it increases your conversion rate by 5%, set your effect size to 5%. With a baseline conversion rate of 10% for the control variant, a test variant would need to show a conversion rate of 10.5% (a 5% relative gain) to count as an improvement. The sample-size estimator shows the number of searches required to detect that improvement with 80% power at a 5% significance level (adjusted in multi-variant testing). When choosing an effect size, consider:- Impact. Consider the smallest change that would make a meaningful improvement in your goals. For example, a 2% relative increase in conversion rate might be significant for you, while another organization might aim for a 5% relative change.
- Historical data. Review past experiments to understand typical variations and set an effect size that’s realistic and achievable based on historical performance. For example, if other changes you have tested result in a 3% relative increase in conversion rate, you might set your effect size to 3%.
- Balance between sensitivity and practicality. Smaller effect sizes, such as 1% to 2%, require larger sample sizes but let you detect subtle changes. Larger effect sizes (for example, 5% to 10%) require smaller samples and are easier to detect but may overlook smaller, yet important, changes.
View the results
Overview page
The overview page lists every A/B test in your with its ID, name, start date, planned duration, and current status (see Test statuses). A/B tests are listed from most recent to oldest, with active tests always at the top. Click the chevron next to Start date to change the sort order.
A/B test details page
- Variant. The tested variant.
- Tracked searches (or Searches for metrics that are calculated regardless of
clickAnalytics). The number of searches served by this variant. - Metric value. The observed value of the metric (for example, CTR, or ATCR).
- Difference. Relative change with respect to the control, expressed as a percentage difference.
- Confidence interval. The statistical range within which the true metric value is expected to lie.
- Confidence rating. Shows how reliable the observed difference is (for example, trending confident, or unconfident). For details, see Confidence. Hover any rating to see the exact p-value.
A/B tests show results in the dashboard within the first hour after you created them,
but metric comparisons won’t show until at least 1 week or 20% of the test duration has elapsed.
This mitigates the risk of drawing inaccurate conclusions due to insufficient data.
Test results are updated every day.
Test statuses
The potential test statuses are:- In progress - Not enough data: the test has started, and there is insufficient data to draw reliable conclusions. Wait for at least one week, or 20% of the test duration, before metrics begin to show. Hover over the badges to see data, but avoid drawing conclusions during this stage.
- In progress: the test has been running for a while, and metrics are being collected and compared.
- Failed: the test couldn’t be created. This can happen due to problems with your index or setting up the test in the Search API. Try to recreate the test or contact the Algolia support team.
- Stopped: the test was stopped manually. Your app is back to normal: index A performs as usual, receiving 100% of search requests. When you stop a test, all associated metadata and metrics are stored, and the test remains visible in the Algolia dashboard. However, the results may be inconclusive.
- Completed: the test has finished. Your app is back to normal: index A performs as usual, receiving 100% of search requests.
How to interpret results
What you consider good or bad is entirely dependent on your needs. Compare the effort and cost of an improvement with its benefits. A 4% improvement in click-through or conversion rate might not be convincing or profitable enough to warrant a change to the structure of your . Given the typically low effort required to adjust settings or modify data, it’s generally advisable to implement any potential improvement. For more information, see How A/B test scores are calculated.Confidence
The confidence for a test is based on the test status and the p-value. A p-value is the probability of observing results at least as extreme as the ones you saw, assuming there is actually no difference between variants. It’s not “confidence” by itself, the false-positive rate, or the probability of replicating the result. A small p-value suggests the observed difference is unlikely to be due to chance alone. A high p-value suggests there isn’t enough evidence to conclude there is a real difference. The different confidence levels are:- Not enough data to interpret: the test has started, and there is insufficient data to draw reliable conclusions on the performance of the variants. Wait for at least a week, or 20% of the test duration, before metrics begin to show. Hover over the badges to see data, but avoid drawing conclusions during this stage.
- No data: the test has been running for a while, but there is no data to compare. This typically occurs when no events are tracked.
- Unconfident: the test has been running for a while, but it’s impossible to tell whether the observed change is representative of the true impact. This could change as more data is collected. Be careful when interpreting these results.
- Trending confident: the test has been running for a while, and it looks like the observed change reflects the true impact. This could change as more data is collected. Be careful when you interpret these results.
- Inconclusive: the test has finished, but the confidence is too low to determine whether the observed change is due to chance. This might be because of insufficient data or high similarity between the variants.
- Confident: the test is complete, and the observed change probably reflects the true impact.
- A Confident result with a large observed decrease in conversion rate suggests that the change will harm conversions. If your goal is to increase conversion rates, avoid this change.
- A Confident result with a large observed increase in conversion rate suggests that the change will improve conversions. If your goal is to increase conversion rates, do implement the change.
- A Trending confident result with a large observed increase in conversion rate suggests that, based on current data, the change will improve conversions. You could implement the change, but the confidence might change later.
- An Inconclusive test means that the impact is uncertain. Ignore the results or interpret them with discretion. Try re-launching the test for a longer duration to collect more data. This increases, but doesn’t guarantee, the likelihood that the results will reach a confident state.
Minimum number of searches
You can stop your test at any point and analyze the results with as little or as much data as you want. Drawing conclusions based on insufficient data and low-confidence test results might lower your overall search performance and lead to unexpected results. The confidence indicator helps ensure the confidence and reliability of test results. Use the confidence indicator to protect you from jumping to conclusions based on skewed or insufficient data.Recommendations
-
Test before going live.
Be wary of breaking anything.
For example, ensure both your test indices work with your UI.
Small changes can break your interface or strongly affect the user experience.
For example:
- Making a change that affects a can cause the facet’s UI logic to fail.
- Changing a simple ranking on index B can make the search results so bad that users of this index have terrible results. This isn’t the purpose of A/B testing. Index B should theoretically be better or at least as good as index A.
- Don’t change your A or B indices during a test. Don’t adjust settings during testing. This pollutes your test results, making them unreliable. If you must update your data, do so synchronously for both indices and, preferably, restart your test. Changing data or settings during a test can break your search experience and undermine your test conclusions.
- Don’t use the same index for several A/B tests. You can’t use the same index in more than one test at the same time. You’ll get an error.
- Make only small changes. The more features you test simultaneously, the harder it is to determine causality.
API clients
In most cases, use the Algolia dashboard to manage A/B tests. However, sometimes, testing with one of the Algolia API clients is beneficial. For example:- You want to run the same test across multiple indices, for example, if you have several websites using the same kind of indices but with different data. The API simplifies the creation of multiple tests.
- You want your backend to trigger tests based on data changes or create feedback loops based on your analytics (as used in machine learning). This is an advanced method of managing product-line changes or industry trends. Monitor outcomes and be ready to revert changes if key metrics regress.
- A/B test creation and deletion:
setSettingson all indices - A/B test analytics:
analyticson all indices