Search by Algolia
Feature Spotlight: Query Rules
product

Feature Spotlight: Query Rules

You’re running an ecommerce site for an electronics retailer, and you’re seeing in your analytics that users keep ...

Jaden Baptista

Technical Writer

An introduction to transformer models in neural networks and machine learning
ai

An introduction to transformer models in neural networks and machine learning

What do OpenAI and DeepMind have in common? Give up? These innovative organizations both utilize technology known as transformer models ...

Vincent Caruana

Sr. SEO Web Digital Marketing Manager

What’s the secret of online merchandise management? Giving store merchandisers the right tools
e-commerce

What’s the secret of online merchandise management? Giving store merchandisers the right tools

As a successful in-store boutique manager in 1994, you might have had your merchandisers adorn your street-facing storefront ...

Catherine Dee

Search and Discovery writer

New features and capabilities in Algolia InstantSearch
engineering

New features and capabilities in Algolia InstantSearch

At Algolia, our business is more than search and discovery, it’s the continuous improvement of site search. If you ...

Haroen Viaene

JavaScript Library Developer

Feature Spotlight: Analytics
product

Feature Spotlight: Analytics

Analytics brings math and data into the otherwise very subjective world of ecommerce. It helps companies quantify how well their ...

Jaden Baptista

Technical Writer

What is clustering?
ai

What is clustering?

Amid all the momentous developments in the generative AI data space, are you a data scientist struggling to make sense ...

Vincent Caruana

Sr. SEO Web Digital Marketing Manager

What is a vector database?
product

What is a vector database?

Fashion ideas for guest aunt informal summer wedding Funny movie to get my bored high-schoolers off their addictive gaming ...

Vincent Caruana

Sr. SEO Web Digital Marketing Manager

Unlock the power of image-based recommendation with Algolia’s LookingSimilar
engineering

Unlock the power of image-based recommendation with Algolia’s LookingSimilar

Imagine you're visiting an online art gallery and a specific painting catches your eye. You'd like to find ...

Raed Chammam

Senior Software Engineer

Empowering Change: Algolia's Global Giving Days Impact Report
algolia

Empowering Change: Algolia's Global Giving Days Impact Report

At Algolia, our commitment to making a positive impact extends far beyond the digital landscape. We believe in the power ...

Amy Ciba

Senior Manager, People Success

Retail personalization: Give your ecommerce customers the tailored shopping experiences they expect and deserve
e-commerce

Retail personalization: Give your ecommerce customers the tailored shopping experiences they expect and deserve

In today’s post-pandemic-yet-still-super-competitive retail landscape, gaining, keeping, and converting ecommerce customers is no easy ...

Vincent Caruana

Sr. SEO Web Digital Marketing Manager

Algolia x eTail | A busy few days in Boston
algolia

Algolia x eTail | A busy few days in Boston

There are few atmospheres as unique as that of a conference exhibit hall: the air always filled with an indescribable ...

Marissa Wharton

Marketing Content Manager

What are vectors and how do they apply to machine learning?
ai

What are vectors and how do they apply to machine learning?

To consider the question of what vectors are, it helps to be a mathematician, or at least someone who’s ...

Catherine Dee

Search and Discovery writer

Why imports are important in JS
engineering

Why imports are important in JS

My first foray into programming was writing Python on a Raspberry Pi to flicker some LED lights — it wasn’t ...

Jaden Baptista

Technical Writer

What is ecommerce? The complete guide
e-commerce

What is ecommerce? The complete guide

How well do you know the world of modern ecommerce?  With retail ecommerce sales having exceeded $5.7 trillion worldwide ...

Vincent Caruana

Sr. SEO Web Digital Marketing Manager

Data is king: The role of data capture and integrity in embracing AI
ai

Data is king: The role of data capture and integrity in embracing AI

In a world of artificial intelligence (AI), data serves as the foundation for machine learning (ML) models to identify trends ...

Alexandra Anghel

Director of AI Engineering

What are data privacy and data security? Why are they  critical for an organization?
product

What are data privacy and data security? Why are they critical for an organization?

Imagine you’re a leading healthcare provider that performs extensive data collection as part of your patient management. You’re ...

Catherine Dee

Search and Discovery writer

Achieving digital excellence: Algolia's insights from the GDS Retail Digital Summit
e-commerce

Achieving digital excellence: Algolia's insights from the GDS Retail Digital Summit

In an era where customer experience reigns supreme, achieving digital excellence is a worthy goal for retail leaders. But what ...

Marissa Wharton

Marketing Content Manager

AI at scale: Managing ML models over time & across use cases
ai

AI at scale: Managing ML models over time & across use cases

Just a few years ago it would have required considerable resources to build a new AI service from scratch. Of ...

Benoit Perrot

VP, Engineering

Looking for something?

facebookfacebooklinkedinlinkedintwittertwittermailmail

Great search should guide users to their needs as quickly as possible, but achieving the right level of relevance for your site is not always straightforward and takes time to get right. With a solid search-as-a-service provider, your default ranking should already be relatively robust. Still, since every business is unique and search changes can impact KPIs and the user experience, you’ll likely want to measure the effect of adding custom ranking factors, new attributes, and more on your overall search relevance. 

With A/B testing, you can leverage data to validate your decisions in real time and make changes to your search with confidence. 

 

What is A/B testing?

A/B testing is the process of concurrently running two different versions of a user interface component while collecting metrics to determine the impact on KPIs. For a search tool, this can be useful in testing changes to relevance algorithms and configurations as there are clear metrics of success to monitor, such as click-through rate and conversions.

There are a few steps to running an A/B test for search relevance:

  1. Pick which metrics and statistical methods are relevant to the test and its outcomes.
  2. Decide the relevant questions to answer, KPIs and metrics to measure and how they relate to your proposed change. This may include a hypothesis that you’re attempting to validate.
  3. Run the two versions of your user interface concurrently to control for variability. The users should be randomly assigned to minimize any statistical noise or bias. Make sure you run the test long enough to collect statistically significant data.
  4. Analyze the test results to determine if your hypothesis was correct and/or how the changes impacted your KPIs. Did the metrics materially increase, did they decline, or did they stay relatively constant? How does this compare to historical data?
  5. Take action! If test results indicate improved KPIs, then roll out the change to all your users. If it was a failure, scrap it and iterate.

 

Why should you test your search?

Search is a fundamental part of the online experience. In fact, Forrester research found that 43% of users on retail websites go directly to the search bar. And, thanks to Google and other global search engines, users expect fast and high-quality results pages on every site they visit. A search experience optimized through A/B testing promotes user engagement by helping them quickly find what they’re looking for and ultimately convert. 

 

A/B testing is a great way to optimize your search experience right from the search box:

  • Perfect relevancy with user input. The purpose of optimizing search and tuning relevancy is to serve users better results. With A/B search testing, your users are directly involved in this process. You can track how users are interacting with your search and make improvements and updates based on real feedback and not just guesswork. Fashion platform Videdressing regularly uses A/B testing to validate product changes for customers shopping from a catalog of over 1 million products.
  • Take advantage of data-driven insights. Users are hard to predict, and A/B test results can help to show exactly how they respond to changes. When you collect enough data for an isolated change against a comparable control group, you can have confidence through statistics that the results are meaningful and make actionable decisions from them.
  • Gain knowledge without risk. A/B search testing allows you to test changes on a representative sample of your users. Since the majority of your users are unaffected, you can be confiderne that revenue and KPIs will not be affected either. 

 

5 ways to A/B test your search

Given that search applies to pretty much all industries, there are a number of ways to experiment with changes to the relevance rules and formulas to see how they impact your KPIs. Here are five important examples of types of tests you can run to improve these metrics:

 

1. See how social proof data affects your search

Leveraging social proof on your site can be highly valuable to your users and an important data point for the business. Showing likes, reviews, and comments on the site can improve visitor’s interest, confidence, and engagement with your offerings. It can also indicate to the business which products or content are particularly popular, helpful, meaningful, or in demand. Therefore, it’s often worth experimenting with ordering of the search results page  by engagement level. 

If the hypothesis is that content with more likes or shares is more relevant to users, then you could show one user segment results based on your existing sort behavior and another user segment results sorted by engagement. Track how this impacts KPIs like click-through rate to determine if the search results are in fact more relevant to users.

 

2. Understand how new searchable attributes affect your search relevance

Indexed content often changes over time as you modify the structure of your documents. If, for example, you add an additional attribute such as a “short product description,” then this content could impact how users explore and engage with it. 

Therefore, you can experiment with including this content in search terms and queries and vary how much the field is boosted relative to other fields. If boosting the new attribute shows improvements in KPIs vs. the existing search algorithm, then it is likely that it is contributing to better relevance of the search results.

 

3. Compare how a query with and without merchandising performs

Search merchandising is a powerful mechanism for promoting products that are important to your business strategies. However, promoting specific products will impact the relevance of search results.

Therefore, before rolling-out a merchandising campaign across a website, it can be valuable to A/B test the change against a control to first understand how it affects KPIs. For example, you might A/B test a merchandising technique such as boosting a category when a user types the category name” to see how this performs. If boosting categories negatively impacts the click-through rate, then you may decide that it is not worth the benefit to roll-out the campaign to the rest of your users. 

 

4. Compare search with and without personalization

Personalization allows you to customize search results to a user based on their historical data. This may include past searches, products added to carts, purchases, or other relevant metrics. As every user has their own unique needs and preferences, this can be valuable for getting them to relevant content faster. However, there are a number of ways to layer on personalization factors and thus it’s important to A/B test the changes to see how they impact KPIs. 

For example, you may hypothesize that showing users products that are in categories related to past purchases will increase their propensity to buy. By A/B testing this theory, you can have confidence that the improvement does in fact increase conversions relative to your existing algorithms—or you can try a different method of personalization instead. 

Decathlon Singapore used A/B testing to gauge the effect of personalization on each search terms and queries. Their tests showed a 50% increase in conversion rate for personalized queries, which enabled them to find the right personalisation balance for different attributes.

 

5. Add and remove some criteria from your ranking formula

There is a fine balance between having too many or too few criteria for your search rankings. Having a lot of parameters may seem like a good idea since you can highly tune the results to your customers’ needs, yet this could lead to having an overly narrow scope and reduce your flexibility to handle a diverse customer base. Conversely, a broad and simplistic ranking formula may be flexible in its scope, yet it could be too general to provide relevant results to your users. There’s no one-size-fits-all approach to optimizing this balance, so A/B testing should be used to determine how adding or removing these criteria affects the relevancy for all of your users.

 

Getting started setting up an A/B test for your search

A/B testing search is important for all websites that have a catalog of content that their users regularly browse and explore. The more relevant the results are that your users see, the more likely they are to stay engaged and continue to use your service. To do so, you’ll need a search-as-a-service provider that provides all of the tools necessary to run effective A/B tests. 

Watch our master class to see how Algolia can help you run a robust and successful A/B test for your search.

 

About the author
Louise Vollaire

Product Marketing Manager

linkedin

Is your site performing as well as it could?

Assess your search experience in less than 5 minutes.

Evaluate your search
Is your site performing as well as it could?

Recommended Articles

Powered byAlgolia Algolia Recommend

How to improve site search with Algolia A/B testing
e-commerce

Loise Mercier

The (almost) ultimate guide to site search
product

Ivana Ivanovic

Senior Content Strategist

Optimizing search performance with A/B testing
product

Nicolas Baissas