Search by Algolia
Feature Spotlight: Query Rules
product

Feature Spotlight: Query Rules

You’re running an ecommerce site for an electronics retailer, and you’re seeing in your analytics that users keep ...

Jaden Baptista

Technical Writer

An introduction to transformer models in neural networks and machine learning
ai

An introduction to transformer models in neural networks and machine learning

What do OpenAI and DeepMind have in common? Give up? These innovative organizations both utilize technology known as transformer models ...

Vincent Caruana

Sr. SEO Web Digital Marketing Manager

What’s the secret of online merchandise management? Giving store merchandisers the right tools
e-commerce

What’s the secret of online merchandise management? Giving store merchandisers the right tools

As a successful in-store boutique manager in 1994, you might have had your merchandisers adorn your street-facing storefront ...

Catherine Dee

Search and Discovery writer

New features and capabilities in Algolia InstantSearch
engineering

New features and capabilities in Algolia InstantSearch

At Algolia, our business is more than search and discovery, it’s the continuous improvement of site search. If you ...

Haroen Viaene

JavaScript Library Developer

Feature Spotlight: Analytics
product

Feature Spotlight: Analytics

Analytics brings math and data into the otherwise very subjective world of ecommerce. It helps companies quantify how well their ...

Jaden Baptista

Technical Writer

What is clustering?
ai

What is clustering?

Amid all the momentous developments in the generative AI data space, are you a data scientist struggling to make sense ...

Vincent Caruana

Sr. SEO Web Digital Marketing Manager

What is a vector database?
product

What is a vector database?

Fashion ideas for guest aunt informal summer wedding Funny movie to get my bored high-schoolers off their addictive gaming ...

Vincent Caruana

Sr. SEO Web Digital Marketing Manager

Unlock the power of image-based recommendation with Algolia’s LookingSimilar
engineering

Unlock the power of image-based recommendation with Algolia’s LookingSimilar

Imagine you're visiting an online art gallery and a specific painting catches your eye. You'd like to find ...

Raed Chammam

Senior Software Engineer

Empowering Change: Algolia's Global Giving Days Impact Report
algolia

Empowering Change: Algolia's Global Giving Days Impact Report

At Algolia, our commitment to making a positive impact extends far beyond the digital landscape. We believe in the power ...

Amy Ciba

Senior Manager, People Success

Retail personalization: Give your ecommerce customers the tailored shopping experiences they expect and deserve
e-commerce

Retail personalization: Give your ecommerce customers the tailored shopping experiences they expect and deserve

In today’s post-pandemic-yet-still-super-competitive retail landscape, gaining, keeping, and converting ecommerce customers is no easy ...

Vincent Caruana

Sr. SEO Web Digital Marketing Manager

Algolia x eTail | A busy few days in Boston
algolia

Algolia x eTail | A busy few days in Boston

There are few atmospheres as unique as that of a conference exhibit hall: the air always filled with an indescribable ...

Marissa Wharton

Marketing Content Manager

What are vectors and how do they apply to machine learning?
ai

What are vectors and how do they apply to machine learning?

To consider the question of what vectors are, it helps to be a mathematician, or at least someone who’s ...

Catherine Dee

Search and Discovery writer

Why imports are important in JS
engineering

Why imports are important in JS

My first foray into programming was writing Python on a Raspberry Pi to flicker some LED lights — it wasn’t ...

Jaden Baptista

Technical Writer

What is ecommerce? The complete guide
e-commerce

What is ecommerce? The complete guide

How well do you know the world of modern ecommerce?  With retail ecommerce sales having exceeded $5.7 trillion worldwide ...

Vincent Caruana

Sr. SEO Web Digital Marketing Manager

Data is king: The role of data capture and integrity in embracing AI
ai

Data is king: The role of data capture and integrity in embracing AI

In a world of artificial intelligence (AI), data serves as the foundation for machine learning (ML) models to identify trends ...

Alexandra Anghel

Director of AI Engineering

What are data privacy and data security? Why are they  critical for an organization?
product

What are data privacy and data security? Why are they critical for an organization?

Imagine you’re a leading healthcare provider that performs extensive data collection as part of your patient management. You’re ...

Catherine Dee

Search and Discovery writer

Achieving digital excellence: Algolia's insights from the GDS Retail Digital Summit
e-commerce

Achieving digital excellence: Algolia's insights from the GDS Retail Digital Summit

In an era where customer experience reigns supreme, achieving digital excellence is a worthy goal for retail leaders. But what ...

Marissa Wharton

Marketing Content Manager

AI at scale: Managing ML models over time & across use cases
ai

AI at scale: Managing ML models over time & across use cases

Just a few years ago it would have required considerable resources to build a new AI service from scratch. Of ...

Benoit Perrot

VP, Engineering

Looking for something?

facebookfacebooklinkedinlinkedintwittertwittermailmail

Relevance is one of the most important attributes of good site search. But it isn’t something you can just flip a switch and turn on. Honing search results for relevance is a process of trial and error as you adjust your back end search parameters to meet your users’ needs and optimize conversions.

We are all familiar with A/B testing your website as a way for you to test how much a particular variable impacts your audience reaction and conversion metrics. And in a previous blog post, we wrote about how to A/B test your site search for relevance.

Today, we’ll dive into how to apply Algolia’s advanced A/B testing feature to your site search.

Why use A/B testing to improve site search

As you already know, A/B testing removes the guesswork from iterations on your relevance by showing you exactly how a particular setting or variable affects your predetermined conversion goals.

For example, you might want to test the way you rank the search results you return to users. Previously, you’ve added the business metric “publication date” in the ranking strategy, but you’re wondering if it might be more effective to use the business metric sales ranking. You think it might drive more conversions, but you want more data before you commit to the change site wide.

In an A/B test, half of the users on your website would be exposed to the original (or control) search experience — the one where your site ranks results by publication date. The other half of your audience will be exposed to the sales ranking system. You can then collect data about the behavior of both sets of users, and have a definitive answer on which setup drives more conversions.

One of the advantages of using Algolia A/B testing is that anyone on your team can create A/B tests entirely from our dashboard, without a single line of code (but it is of course available from the API if that’s what you prefer to do), and on any device.

4 great A/B testing example scenarios

What can you test with site search A/B testing? Here are some ideas:

  • Tweak your custom rankings. Let’s say you have an e-commerce website where your items are currently sorted by popularity, but you are not actually certain that this is the best way to show results. Test the effectiveness of sorting them by publication date. A/B testing will let you test these two different custom ranking options against each other to see which one drives more sales.
  • Test the impact of personalization. Once you start using personalization, you may want to assess the impact it has on your conversion and click-through rates. A/B testing is a great way to do this.
  • A/B test the impact of synonyms. Test the impact of two different sets of synonyms on the same set of records and see which one actually performs better in terms of relevance.
  • A/B test two sets of Query Rules. Test the impact of two different sets of Query Rules — two merchandising strategies, for example — to see which one actually performs better.

How does A/B testing actually work?

Let’s look at a practical application of A/B testing with Algolia: an easy, six-step process. (Note: you’ll need to set up Click Analytics first in order to capture click & conversion events).

1. Create a replica

Let’s go to your search dashboard. You can already see there is an index created, with defined custom ranking – # of sales per item (aka Sales ranking)

Creating a replica index for A/B testing site search

But, you are not sure this is the best ranking strategy, so you’d like to check it against ranking by publication date.

When you A/B test, you’re testing two search indices against each other, so the first thing you need to do is create a replica — aka a copy of your search index. To create a replica, go to the Indices section of the left-hand toolbar and click the “Replicas” option. From there, you can select the option to “Create Replica Index.”

The replica will be synchronized with your main index, so any changes to the main index will be forwarded to your replica. They are identical apart from the changes you manually make to the replica.

Two replicas are identical except the changes you manually make to the replica.

2. Configure your replica

configuring your replica

Once you’ve created your replica, it’s time to configure it with the factor you are testing. In this case, we’ve defined a different custom ranking based on publication date instead of sales ranking.

3. Create a new A/B test

Once you’ve configured your replica, go to the A/B testing tab on the left-hand toolbar. Click on “New test” to create your new A/B test.

creating a new AB test

4. Pick your variants

Once you create a new test, you’ll have the option to pick the two variants you plan to test. The first variant should be your main index or your control. The second will be the replica you just created. It’s a good idea to provide a description for each variant for later reference, such as “custom ranking based on sales rank” and “custom ranking based on publication date.”

Picking your A/B test variants

5. Define your traffic split

Underneath the variants, define the percentage traffic split between the two variables. An ideal split is 50/50, but if you have a high-traffic website (more than 100,000 page views a month), you can define an unequal traffic split and still see statistically significant results.

6. Define the test duration

Ideally, we want an A/B test to run for what we call two business cycles, which takes into account short-term seasonality effects. For an e-commerce website, a business cycle would mean one week. So, in this case, I want my test to run for 14 days.

Once you’ve defined these parameters, simply press “Create.”

How to interpret the results

The most important part of A/B testing is analyzing the results to determine which variant was most successful. Algolia’s results panel is clearly structured to help you make that decision:

Interpreting AB test results

In the results panel, you’ll see identifying information about the test: the name, the scenarios (the two indices you tested), and the status. The test will either be “Running,” “Stopped,” “Finished,” or “Failed.” You can click on the analytics bar next to each scenario to see the individual metrics for each.

Here are the other important components of the test panel:

  • Traffic split. This reflects the volume of users assigned to one index or the other.
  • Number of users. This tells you how many users were exposed to each variable. If a large volume of users were tested, this tells you the results are significant.
  • CTR. This is the click-through rate or the number of users who clicked on results over the total number of searches, times a hundred. A click indicates the user found what they were looking for.
  • Conversion. This is the conversion rate — the number of conversion events (like sales or downloads) over the total number of searches, times a hundred.
  • Confidence score. This tells you how confident Algolia is that the results are a consequence of the configuration change and not random chance. The confidence score should be higher than 95% before you analyze the results.

Once your test reaches a 95% or greater confidence score and your two-cycle time period has passed, you can analyze the results to see which index generated more clicks and conversions.

In the A/B testing example above, you can see that the sales rank custom setup (Scenario B) performed better with a 5.2% uplift on CTR and a 4% increase in conversion rate. The confidence score for both is excellent, meaning there were enough searches and events to draw the conclusion that B yields better results. Once the test is over, you can apply the winning configuration to the main index.

In this test, the numbers demonstrate that ranking search results by sales rank leads to more conversions rather than ranking by publication date.

Best practices for A/B testing site search

A/B testing is a scientific process, so it’s important that you consistently follow a few guidelines to ensure the most accurate results.

Set a proper user token

User tokens are the user identifiers that connect search events with eventual conversions. IP addresses alone are not sufficient for accurate tracking and can lead to incorrect results. We highly recommend that you explicitly use a user token for the most accurate results. If you aren’t already using them, you can generate them with Algolia’s insights cookie.

Use the same user token for searches and for click events

It is very important to use the same user token for searches and for click events because that’s how the A/B test will connect the users with the events that those users performed. If you’re using two different user tokens, then the A/B test won’t work.

Disable A/B test (and analytics) for automated search agents or bots

If your website uses search automators (for example, if you have an option where users can subscribe to alerts when their search returns new results), then exclude those search agents from your A/B test. If you don’t, you’ll end up with a single “user” who is performing a vast number of searches but never clicks or converts. This can severely bias one side of your test results. Make sure automations are excluded from both your testing and your analytics setups.

Wait until the end of your test before relying on the confidence score

The confidence score is calculated based on the number of searches conducted — so you may actually reach a high confidence level within just a few hours of starting your test. That doesn’t mean these results are correct, though. It’s important to leave your test running for two full business cycles to account for all the seasonality effects that might change your results. For example, people search differently on the weekends versus weekdays.

Test a single hypothesis at once

Only change one search element at a time so that you can clearly test the effects of that change on your users’ behavior. Trying to run more than one A/B test at a time can lead to confusing and unclear results. How do you know which change led to which behavior?

Don’t make changes once the test is live

Once the test is running, don’t make any changes to either of the test indices, the traffic allocations, or the test goals. Changing variables mid-experiment will make the results irrelevant, as it will be impossible to tell which modification impacted user behavior.

Make search decisions with confidence

Improving relevance is the shortcut to improving your site search functionality. You can experiment to hone those results, but you don’t want to tank your conversions or customer satisfaction in the process.

Just like you A/B test other components of your website, you should A/B test your site search to eliminate the guesswork and zero in on the variables that make your results useful and effective. Test only a subset of the population to avoid costly mistakes — this will also give you confidence when you go to make major changes.

To learn more about A/B testing with Algolia and to see how some major companies have used it to improve their conversions, watch our A/B testing Master Class, or visit our A/B testing product page.

About the author
Loise Mercier

Master Class: A/B testing with Algolia

Join Loïse Mercier, Customer Success Specialist, and Rémy Zeiss, Product Manager, for a short webinar jam-packed with information on A/B testing in general, and with Algolia.

Master Class: A/B testing with AlgoliaWatch webinar

Recommended Articles

Powered byAlgolia Algolia Recommend

5 ways to A/B test your search for relevance
ux

Louise Vollaire

Product Marketing Manager

Optimizing search performance with A/B testing
product

Nicolas Baissas

How to design your merchandising (and searchandising) stategies for success
e-commerce

Peter Villani

Sr. Tech & Business Writer