Step 1: Add a selected custom attribute (for example, salesRank attribute) to your main catalog index (this is index A in your A/B test)

For example: numberOfLikes attribute, will allow sorting the search results based on the “likes” data that was gathered from site users. Alternatively, salesRank attribute will allow sorting search results based on the internally gathered sales data.

For step by step instructions refer to the guide “Configuring Custom Ranking for Business Relevance”

Step 2: Create index B as a replica of A. 

1. Load the dashboard section “Index”

2. Select the correct index

3. In the top left section click on the “New” button

4. In the drop down menu select “Replica”

1. In the pop-up window “Create a new replica” type in the replica name (Index B)

2. Select the the type: “standard” or “virtual”

3. Review before saving the replica index

   

Step 3: Adjust index B’s settings by sorting it’s records based on the selected custom attribute (for example, salesRank attribute)

1. Load the dashboard section “Index”

2. Verify the correct replica index is selected

3. In the top navigation menu select “Configuration” tab

4. Click on “Ranking and Sorting” 

5. Click on the button “Add sort-by attribute” and select the relevant attribute. (For example: salesRank)

6. Note: the ranking attributes need to be pre-configured by the data engineering team.

   

7. Once you add a sort-by attribute, it will appear on the top of the “Ranking and sorting” list. Select the desired order for this attribute: ascending or descending and save your changes. 

8. Note: While each ranking and sorting attribute’s position can be changed by dragging and dropping it higher or lower on the list, it is not recommended to do so.

9. Review the changes before saving

   

Step 4: Create an A/B test

Tips for creating A/B test:

• Assign your test a self explanatory and meaningful name. For example: “Test new ranking with salesRank”.
• Let this test run for 30 days, to be sure to get enough data and a good variety of searches, so you set the date parameters accordingly.
• In most cases it’s fine to set the test to use a 50/50 split. If you are concerned that the new version might have a negative impact you can show it to a smaller percentage of customers. The trade off here is that a smaller sample means your test will take longer to reach statistical significance. This is particularly true if you don’t have a huge volume of visitors to your site.
• When your test reaches 95% confidence or greater, you can see whether your change improves your search, and whether the improvement is large enough to justify implementational cost.

1. Load the dashboard section “A/B testing”

2. Verify the correct application is selected

3. In the top right section click on the “New test” button

   

4. Select an informative and self explanatory name for the test

5. Select “Variant A” - an index currently used for the live search

6. Select “Variant B” - a replica of index chosen as a variant A with the configuration changes applied to it that you would like to test.

7. Define the percentage of traffic that will be split and directed to each variant

8. Set a test duration. The recommended minimum duration is 30 days.

9. Review the changes before saving

   

10. The new A/B test will appear on the A/B testing overview section of the dashboard, where you can view and analyse it once the user data will start aggregating

    

Step 1: Add a new custom attribute (for example, short_description attribute) to your main catalog index (this is index A in your A/B test). You will want to test if setting a new attribute (in this example, a new short description to each of your records) as a searchable attribute will improve your relevance.

Tip: Because you are testing a search time setting for your index, you can use A as both indices in your test.

1. Load the dashboard sectionIndex
2. Select the correct index
3. In the top navigation menu select “Configuration” tab
4. Select on “Searchable attributes”, then click on “Add a Searchable Attribute”

Step 2: Create and run an A/B test

1. Load the dashboard section “A/B testing”

2. Verify the correct application is selected

3. In the top right section click on the “New test” button

   

4. Select an informative and self explanatory name for the test. For example, “Testing the new short description”

5. Create your A/B test with index A as both variants

6. Set the selected searchable attribute (in this example, short_description) as the varying searchable attribute.

7. Define the percentage of traffic that will be split and directed to each variant. For example, give B only 30% usage (70/30) - because of the uncertainty: you’d rather not risk degrading an already good search with an untested attribute.

8. Set a test duration. The recommended minimum duration is 30 days.

9. Review the changes before saving

   

10. The new A/B test will appear on the A/B testing overview section of the dashboard, where you can view and analyse it once the user data will start aggregating

Once you reach 95% confidence, you can judge the improvement and the cost of implementation to see whether this change is beneficial.

Step 1: Create Rules for your main catalog index (this is A in the test). 

For a step-by-step instruction, please refer to “Category merchandising” and “Search merchandising” sections of this guide.

For example: promote your new Kindle product record using rules

Step 2: 

1. Load the dashboard section “A/B testing”

2. Verify the correct application is selected

3. In the top right section click on the “New test” button

   

4. Select an informative and self explanatory name for the test. For example, “Testing the newly-released Kindle merchandising”

5. Create your A/B test with index A as both variants

   

6. As the varying query parameter, add enableRules.  

   

7. Define the percentage of traffic that will be split and directed to each variant. For example, give B only 30% usage (70/30) - because of the uncertainty: you’d rather not risk degrading an already good search with an untested attribute.

8. Set a test duration. The recommended minimum duration is 30 days.

9. Review the changes before saving

   

10. The new A/B test will appear on the A/B testing overview section of the dashboard, where you can view and analyse it once the user data will start aggregating

Once you reach 95% confidence, you can judge the improvement and the cost of implementation to see whether this change is beneficial.