Resources Relevance & Tuning

Implementing A/B Testing

Create two alternative search experiences with unique settings, put them both live, and see which one performs best.

Set up 

  1. Set up Click and Conversion Analytics. (This step is performed by the engineering team)
  2. Set up index or query for A/B testing. There are two ways of creating A/B tests:
    • The first is to show results from the same index with different parameter added. This allows you to test things like enabling personalization, re-ranking, rules, facet re-ordering and many other settings. 
    • The second is to show results form a different index. If you want to test changes to the base relevance like which attributes are searchable, or which custom ranking factors to use, you will need a separate index, usually a replica. 
  3. Run the A/B test:
    • Use the A/B test tab of the Dashboard to create your test
    • Start the A/B test.
  4. Access A/B testing analytics on the Dashboard to view detailed analytics for each variant independently

Step 1: Add a selected custom attribute (for example, salesRank attribute) to your main catalog index (this is index A in your A/B test)

For example: numberOfLikes attribute, will allow sorting the search results based on the “likes” data that was gathered from site users. Alternatively, salesRank attribute will allow sorting search results based on the internally gathered sales data.

For step by step instructions refer to the guide “Configuring Custom Ranking for Business Relevance”

AB Testing

Step 2: Create index B as a replica of A. 

  1. Load the dashboard section “Index”
  2. Select the correct index
  3. In the top left section click on the “New” button
  4. In the drop down menu select “Replica”
    Replica
  5. In the pop-up window “Create a new replica” type in the replica name (Index B)
  6. Select the the type: “standard” or “virtual”
  7. Review before saving the replica index
    review before saving

Step 3: Adjust index B’s settings by sorting it’s records based on the selected custom attribute (for example, salesRank attribute)

  1. Load the dashboard section “Index”
  2. Verify the correct replica index is selected
  3. In the top navigation menu select “Configuration” tab
  4. Click on “Ranking and Sorting” 
  5. Click on the button “Add sort-by attribute” and select the relevant attribute. (For example: salesRank)
  6. Note: the ranking attributes need to be pre-configured by the data engineering team.
    Add sort-by attribute
  7. Once you add a sort-by attribute, it will appear on the top of the “Ranking and sorting” list. Select the desired order for this attribute: ascending or descending and save your changes. 
  8. Note: While each ranking and sorting attribute’s position can be changed by dragging and dropping it higher or lower on the list, it is not recommended to do so.
  9. Review the changes before saving
    review changes before saving

Step 4: Create an A/B test

Tips for creating A/B test:

  • Assign your test a self explanatory and meaningful name. For example: “Test new ranking with salesRank”.
  • Let this test run for 30 days, to be sure to get enough data and a good variety of searches, so you set the date parameters accordingly.
  • In most cases it’s fine to set the test to use a 50/50 split. If you are concerned that the new version might have a negative impact you can show it to a smaller percentage of customers. The trade off here is that a smaller sample means your test will take longer to reach statistical significance. This is particularly true if you don’t have a huge volume of visitors to your site.
  • When your test reaches 95% confidence or greater, you can see whether your change improves your search, and whether the improvement is large enough to justify implementational cost.
  1. Load the dashboard section “A/B testing”
  2. Verify the correct application is selected
  3. In the top right section click on the “New test” button
    New test button
  4. Select an informative and self explanatory name for the test
  5. Select “Variant A” - an index currently used for the live search
  6. Select “Variant B” - a replica of index chosen as a variant A with the configuration changes applied to it that you would like to test.
  7. Define the percentage of traffic that will be split and directed to each variant
  8. Set a test duration. The recommended minimum duration is 30 days.
  9. Review the changes before saving
    review the change before saving
  10. The new A/B test will appear on the A/B testing overview section of the dashboard, where you can view and analyse it once the user data will start aggregating
    New ab test appears

Step 1: Add a new custom attribute (for example, short_description attribute) to your main catalog index (this is index A in your A/B test). You will want to test if setting a new attribute (in this example, a new short description to each of your records) as a searchable attribute will improve your relevance.

Tip: Because you are testing a search time setting for your index, you can use A as both indices in your test.

  1. Load the dashboard section “Index”
  2. Select the correct index
  3. In the top navigation menu select “Configuration” tab
  4. Select on “Searchable attributes”, then click on “Add a Searchable Attribute”
    Add searchable attribute

Step 2: Create and run an A/B test

  1. Load the dashboard section “A/B testing”
  2. Verify the correct application is selected
  3. In the top right section click on the “New test” button
    New test button
  4. Select an informative and self explanatory name for the test. For example, “Testing the new short description”
  5. Create your A/B test with index A as both variants
  6. Set the selected searchable attribute (in this example, short_description) as the varying searchable attribute.
  7. Define the percentage of traffic that will be split and directed to each variant. For example, give B only 30% usage (70/30) - because of the uncertainty: you’d rather not risk degrading an already good search with an untested attribute.
  8. Set a test duration. The recommended minimum duration is 30 days.
  9. Review the changes before saving
    review changes
  10. The new A/B test will appear on the A/B testing overview section of the dashboard, where you can view and analyse it once the user data will start aggregating

Once you reach 95% confidence, you can judge the improvement and the cost of implementation to see whether this change is beneficial.