Algolia Academy
Learn more about: A/B Testing: Optimizing Performance
Advantages of A/B testing
A/B testing helps you:- Limit disruption: test changes to an or search configuration with a share of your live traffic before applying them to all users.
- Base decisions on user behavior: use click and conversion events to compare how configurations affect engagement.
Set up A/B tests
After you configure your app to send click and conversion events, you can create and manage A/B tests in the Algolia dashboard without writing more code.Collect click and conversion events
Configure your app to send click and conversion events before you start a test. These events let you compare clicks and conversions for each variant.Set up indices or search parameters
The setup depends on whether you want to compare index settings or search parameters:- Test different index settings (applies to any parameter you can use only in the
settingsscope). This requires creating replicas, which increases your record count. - Test different search parameters (no index setup required).
Run A/B tests
After you prepare your indices or search parameters, create and launch an A/B test. Review results using the statistical method you selected. For frequentist tests, wait until the planned end before making a decision. For Bayesian tests, see Bayesian experimentation. Use what you learn to plan your next test.A/B testing in the Algolia dashboard
In the Algolia dashboard, select your Algolia application. On the left sidebar, select Search, then A/B Testing. The overview page lists each test’s status, start date, and duration. Select a test to see the metrics for each variant. Click View analytics to open search analytics for the test’s variants.Example A/B tests
The following examples compare custom ranking, searchable attributes, and rules. You can also test other search-time settings, such as typo tolerance and optional filters.Test custom ranking by number of likes
Your users can like items such as music, films, and blog posts. To test whether custom ranking by number of likes improves your search results:- On the A/B Testing page, click Create test and select Index vs. Index test. Select the main index as the control and the replica as variant B.
- Add a numeric
number_of_likesattribute to the in your main catalog index. - In the replica’s settings, add
desc(number_of_likes)tocustomRanking. Keep the other settings unchanged. - Name your test “Test new ranking with number of likes”.
- Assign 10% of traffic to variant B and 90% to the control to limit how many users see the ranking change.
- If the sample size estimator is available for this test type, use it to choose a duration. Otherwise, set the duration based on your traffic and the effect you want to detect.
- Name the test
Test ranking by number of likes. Review the configuration and click Launch test. - Review the results using the test’s statistical method. Compare any improvement with the cost of changing the ranking before adopting it.
Test a searchable short description
Your records include ashort_description attribute.
This example assumes your main index has an explicit searchableAttributes list that doesn’t include it.
To test whether searching this attribute improves relevance, follow these steps:
- Use your main index as the control.
Don’t change its
searchableAttributessetting. - Create a standard replica for variant B.
Add
short_descriptionto the end of the replica’ssearchableAttributeslist. Keep the other settings unchanged. - On the A/B Testing page, click Create test and select Index vs. Index test. Select the main index as the control and the replica as variant B.
- Assign 30% of traffic to variant B and 70% to the control to limit how many users search the added attribute.
- If the sample size estimator is available for this test type, use it to choose a duration. Otherwise, set the duration based on your traffic and the effect you want to detect.
- Name the test
Test searchable short descriptions. Review the configuration and click Launch test. - Review the results using the test’s statistical method.
Compare any improvement with the cost of making
short_descriptionsearchable.
Compare searches with and without rules
Use this test to compare searches that apply index rules with searches that don’t. This measures the combined effect of the rules, not just the effect of the promotion in this example. In this example, you promote an iPhone model for queries containingapple, iphone, or mobile.
To compare the effect of rules on clicks and conversions, follow these steps:
- In your main index, set
enableRulestotrue. Use this index as the control. - Create a rule on the main index that promotes the iPhone record to the first position for the target queries.
- On the A/B Testing page, click Create test and select Index configurations. Use the main index for the control and variant B.
- For variant B, click Add Query Parameter. Next to Enable Rules, turn off Tested value.
- Assign 30% of traffic to variant B and 70% to the control to limit how many users search without rules.
- If the sample size estimator is available for this test type, use it to choose a duration. Otherwise, set the duration based on your traffic and the effect you want to detect.
- Name the test
Test searches with and without rules. Review the configuration and click Launch test. - Review the results using the test’s statistical method. Compare clicks and conversions between the control and variant B to assess the effect of rules.