Search by Algolia
Introducing new developer-friendly pricing
algolia

Introducing new developer-friendly pricing

Hey there, developers! At Algolia, we believe everyone should have the opportunity to bring a best-in-class search experience ...

Nick Vlku

VP of Product Growth

What is online visual merchandising?
e-commerce

What is online visual merchandising?

Eye-catching mannequins. Bright, colorful signage. Soothing interior design. Exquisite product displays. In short, amazing store merchandising. For shoppers in ...

Catherine Dee

Search and Discovery writer

Introducing the new Algolia no-code data connector platform
engineering

Introducing the new Algolia no-code data connector platform

Ingesting data should be easy, but all too often, it can be anything but. Data can come in many different ...

Keshia Rose

Staff Product Manager, Data Connectivity

Customer-centric site search trends
e-commerce

Customer-centric site search trends

Everyday there are new messages in the market about what technology to buy, how to position your company against the ...

Piyush Patel

Chief Strategic Business Development Officer

What is online retail merchandising? An introduction
e-commerce

What is online retail merchandising? An introduction

Done any shopping on an ecommerce website lately? If so, you know a smooth online shopper experience is not optional ...

Vincent Caruana

Sr. SEO Web Digital Marketing Manager

5 considerations for Black Friday 2023 readiness
e-commerce

5 considerations for Black Friday 2023 readiness

It’s hard to imagine having to think about Black Friday less than 4 months out from the previous one ...

Piyush Patel

Chief Strategic Business Development Officer

How to increase your sales and ROI with optimized ecommerce merchandising
e-commerce

How to increase your sales and ROI with optimized ecommerce merchandising

What happens if an online shopper arrives on your ecommerce site and: Your navigation provides no obvious or helpful direction ...

Catherine Dee

Search and Discovery writer

Mobile search UX best practices, part 3: Optimizing display of search results
ux

Mobile search UX best practices, part 3: Optimizing display of search results

In part 1 of this blog-post series, we looked at app interface design obstacles in the mobile search experience ...

Vincent Caruana

Sr. SEO Web Digital Marketing Manager

Mobile search UX best practices, part 2: Streamlining search functionality
ux

Mobile search UX best practices, part 2: Streamlining search functionality

In part 1 of this series on mobile UX design, we talked about how designing a successful search user experience ...

Vincent Caruana

Sr. SEO Web Digital Marketing Manager

Mobile search UX best practices, part 1: Understanding the challenges
ux

Mobile search UX best practices, part 1: Understanding the challenges

Welcome to our three-part series on creating winning search UX design for your mobile app! This post identifies developer ...

Vincent Caruana

Sr. SEO Web Digital Marketing Manager

Teaching English with Zapier and Algolia
engineering

Teaching English with Zapier and Algolia

National No Code Day falls on March 11th in the United States to encourage more people to build things online ...

Alita Leite da Silva

How AI search enables ecommerce companies to boost revenue and cut costs
ai

How AI search enables ecommerce companies to boost revenue and cut costs

Consulting powerhouse McKinsey is bullish on AI. Their forecasting estimates that AI could add around 16 percent to global GDP ...

Michelle Adams

Chief Revenue Officer at Algolia

What is digital product merchandising?
e-commerce

What is digital product merchandising?

How do you sell a product when your customers can’t assess it in person: pick it up, feel what ...

Catherine Dee

Search and Discovery writer

Scaling marketplace search with AI
ai

Scaling marketplace search with AI

It is clear that for online businesses and especially for Marketplaces, content discovery can be especially challenging due to the ...

Bharat Guruprakash

Chief Product Officer

The changing face of digital merchandising
e-commerce

The changing face of digital merchandising

This 2-part feature dives into the transformational journey made by digital merchandising to drive positive ecommerce experiences. Part 1 ...

Reshma Iyer

Director of Product Marketing, Ecommerce

What’s a convolutional neural network and how is it used for image recognition in search?
ai

What’s a convolutional neural network and how is it used for image recognition in search?

A social media user is shown snapshots of people he may know based on face-recognition technology and asked if ...

Catherine Dee

Search and Discovery writer

What’s organizational knowledge and how can you make it accessible to the right people?
product

What’s organizational knowledge and how can you make it accessible to the right people?

How’s your company’s organizational knowledge holding up? In other words, if an employee were to leave, would they ...

Catherine Dee

Search and Discovery writer

Adding trending recommendations to your existing e-commerce store
engineering

Adding trending recommendations to your existing e-commerce store

Recommendations can make or break an online shopping experience. In a world full of endless choices and infinite scrolling, recommendations ...

Ashley Huynh

Looking for something?

5 ways to A/B test your search for relevance
facebookfacebooklinkedinlinkedintwittertwittermailmail

Great search should guide users to their needs as quickly as possible, but achieving the right level of relevance for your site is not always straightforward and takes time to get right. With a solid search-as-a-service provider, your default ranking should already be relatively robust. Still, since every business is unique and search changes can impact KPIs and the user experience, you’ll likely want to measure the effect of adding custom ranking factors, new attributes, and more on your overall search relevance. 

With A/B testing, you can leverage data to validate your decisions in real time and make changes to your search with confidence. 

 

What is A/B testing?

A/B testing is the process of concurrently running two different versions of a user interface component while collecting metrics to determine the impact on KPIs. For a search tool, this can be useful in testing changes to relevance algorithms and configurations as there are clear metrics of success to monitor, such as click-through rate and conversions.

There are a few steps to running an A/B test for search relevance:

  1. Pick which metrics and statistical methods are relevant to the test and its outcomes.
  2. Decide the relevant questions to answer, KPIs and metrics to measure and how they relate to your proposed change. This may include a hypothesis that you’re attempting to validate.
  3. Run the two versions of your user interface concurrently to control for variability. The users should be randomly assigned to minimize any statistical noise or bias. Make sure you run the test long enough to collect statistically significant data.
  4. Analyze the test results to determine if your hypothesis was correct and/or how the changes impacted your KPIs. Did the metrics materially increase, did they decline, or did they stay relatively constant? How does this compare to historical data?
  5. Take action! If test results indicate improved KPIs, then roll out the change to all your users. If it was a failure, scrap it and iterate.

 

Why should you test your search?

Search is a fundamental part of the online experience. In fact, Forrester research found that 43% of users on retail websites go directly to the search bar. And, thanks to Google and other global search engines, users expect fast and high-quality results pages on every site they visit. A search experience optimized through A/B testing promotes user engagement by helping them quickly find what they’re looking for and ultimately convert. 

 

A/B testing is a great way to optimize your search experience right from the search box:

  • Perfect relevancy with user input. The purpose of optimizing search and tuning relevancy is to serve users better results. With A/B search testing, your users are directly involved in this process. You can track how users are interacting with your search and make improvements and updates based on real feedback and not just guesswork. Fashion platform Videdressing regularly uses A/B testing to validate product changes for customers shopping from a catalog of over 1 million products.
  • Take advantage of data-driven insights. Users are hard to predict, and A/B test results can help to show exactly how they respond to changes. When you collect enough data for an isolated change against a comparable control group, you can have confidence through statistics that the results are meaningful and make actionable decisions from them.
  • Gain knowledge without risk. A/B search testing allows you to test changes on a representative sample of your users. Since the majority of your users are unaffected, you can be confiderne that revenue and KPIs will not be affected either. 

 

5 ways to A/B test your search

Given that search applies to pretty much all industries, there are a number of ways to experiment with changes to the relevance rules and formulas to see how they impact your KPIs. Here are five important examples of types of tests you can run to improve these metrics:

 

1. See how social proof data affects your search

Leveraging social proof on your site can be highly valuable to your users and an important data point for the business. Showing likes, reviews, and comments on the site can improve visitor’s interest, confidence, and engagement with your offerings. It can also indicate to the business which products or content are particularly popular, helpful, meaningful, or in demand. Therefore, it’s often worth experimenting with ordering of the search results page  by engagement level. 

If the hypothesis is that content with more likes or shares is more relevant to users, then you could show one user segment results based on your existing sort behavior and another user segment results sorted by engagement. Track how this impacts KPIs like click-through rate to determine if the search results are in fact more relevant to users.

 

2. Understand how new searchable attributes affect your search relevance

Indexed content often changes over time as you modify the structure of your documents. If, for example, you add an additional attribute such as a “short product description,” then this content could impact how users explore and engage with it. 

Therefore, you can experiment with including this content in search terms and queries and vary how much the field is boosted relative to other fields. If boosting the new attribute shows improvements in KPIs vs. the existing search algorithm, then it is likely that it is contributing to better relevance of the search results.

 

3. Compare how a query with and without merchandising performs

Search merchandising is a powerful mechanism for promoting products that are important to your business strategies. However, promoting specific products will impact the relevance of search results.

Therefore, before rolling-out a merchandising campaign across a website, it can be valuable to A/B test the change against a control to first understand how it affects KPIs. For example, you might A/B test a merchandising technique such as boosting a category when a user types the category name” to see how this performs. If boosting categories negatively impacts the click-through rate, then you may decide that it is not worth the benefit to roll-out the campaign to the rest of your users. 

 

4. Compare search with and without personalization

Personalization allows you to customize search results to a user based on their historical data. This may include past searches, products added to carts, purchases, or other relevant metrics. As every user has their own unique needs and preferences, this can be valuable for getting them to relevant content faster. However, there are a number of ways to layer on personalization factors and thus it’s important to A/B test the changes to see how they impact KPIs. 

For example, you may hypothesize that showing users products that are in categories related to past purchases will increase their propensity to buy. By A/B testing this theory, you can have confidence that the improvement does in fact increase conversions relative to your existing algorithms—or you can try a different method of personalization instead. 

Decathlon Singapore used A/B testing to gauge the effect of personalization on each search terms and queries. Their tests showed a 50% increase in conversion rate for personalized queries, which enabled them to find the right personalisation balance for different attributes.

 

5. Add and remove some criteria from your ranking formula

There is a fine balance between having too many or too few criteria for your search rankings. Having a lot of parameters may seem like a good idea since you can highly tune the results to your customers’ needs, yet this could lead to having an overly narrow scope and reduce your flexibility to handle a diverse customer base. Conversely, a broad and simplistic ranking formula may be flexible in its scope, yet it could be too general to provide relevant results to your users. There’s no one-size-fits-all approach to optimizing this balance, so A/B testing should be used to determine how adding or removing these criteria affects the relevancy for all of your users.

 

Getting started setting up an A/B test for your search

A/B testing search is important for all websites that have a catalog of content that their users regularly browse and explore. The more relevant the results are that your users see, the more likely they are to stay engaged and continue to use your service. To do so, you’ll need a search-as-a-service provider that provides all of the tools necessary to run effective A/B tests. 

Watch our master class to see how Algolia can help you run a robust and successful A/B test for your search.

 

About the author
Louise Vollaire

Product Marketing Manager

linkedin

Is your site performing as well as it could?

Assess your search experience in less than 5 minutes.

Evaluate your search
Is your site performing as well as it could?

Recommended Articles

Powered byAlgolia Algolia Recommend

How to improve site search with Algolia A/B testing
e-commerce

Loise Mercier

The (almost) ultimate guide to site search
product

Ivana Ivanovic

Senior Content Strategist

Optimizing search performance with A/B testing
product

Nicolas Baissas