Search by Algolia
8 ways to use merchandising data to boost your online store ROI
e-commerce

8 ways to use merchandising data to boost your online store ROI

New year, new goals. Sounds positive, but looking at your sales data, your revenue and profit aren’t so hot ...

John Stewart

VP, Corporate Communications and Brand

Algolia DocSearch + Astro Starlight
engineering

Algolia DocSearch + Astro Starlight

What is Astro Starlight? If you're building a documentation site, your content needs to be easy to write and ...

Jaden Baptista

Technical Writer

What role does AI play in recommendation systems and engines?
ai

What role does AI play in recommendation systems and engines?

You put that in your cart. How about this cool thing to go with it? You liked that? Here are ...

Catherine Dee

Search and Discovery writer

How AI can help improve your user experience
ux

How AI can help improve your user experience

They say you get one chance to make a great first impression. With visual design on ecommerce web pages, this ...

Jon Silvers

Director, Digital Marketing

Keeping your Algolia search index up to date
product

Keeping your Algolia search index up to date

When creating your initial Algolia index, you may seed the index with an initial set of data. This is convenient ...

Jaden Baptista

Technical Writer

Merchandising in the AI era
e-commerce

Merchandising in the AI era

For merchandisers, every website visit is an opportunity to promote products to potential buyers. In the era of AI, incorporating ...

Tariq Khan

Director of Content Marketing

Debunking the most common AI myths
ai

Debunking the most common AI myths

ARTIFICIAL INTELLIGENCE CAN’T BE TRUSTED, shouts the headline on your social media newsfeed. Is that really true, or is ...

Vincent Caruana

Senior Digital Marketing Manager, SEO

How AI can benefit the retail industry
ai

How AI can benefit the retail industry

Artificial intelligence is on a roll. It’s strengthening healthcare diagnostics, taking on office grunt work, helping banks combat fraud ...

Catherine Dee

Search and Discovery writer

How ecommerce AI is reshaping business
e-commerce

How ecommerce AI is reshaping business

Like other modern phenomena such as social media, artificial intelligence has landed on the ecommerce industry scene with a giant ...

Vincent Caruana

Senior Digital Marketing Manager, SEO

AI-driven smart merchandising: what it is and why your ecommerce store needs it
ai

AI-driven smart merchandising: what it is and why your ecommerce store needs it

Do you dream of having your own personal online shopper? Someone familiar and fun who pops up every time you ...

Catherine Dee

Search and Discovery writer

NRF 2024: A cocktail of inspiration and innovation
e-commerce

NRF 2024: A cocktail of inspiration and innovation

Retail’s big show, NRF 2024, once again brought together a wide spectrum of practitioners focused on innovation and transformation ...

Reshma Iyer

Director of Product Marketing, Ecommerce

How AI-powered personalization is transforming the user and customer experience
ai

How AI-powered personalization is transforming the user and customer experience

In a world of so many overwhelming choices for consumers, how can you best engage with the shoppers who visit ...

Vincent Caruana

Senior Digital Marketing Manager, SEO

Unveiling the future: Algolia’s AI revolution at NRF Retail Big Show
algolia

Unveiling the future: Algolia’s AI revolution at NRF Retail Big Show

Get ready for an exhilarating journey into the future of retail as Algolia takes center stage at the NRF Retail ...

John Stewart

VP Corporate Marketing

How to master personalization with AI
ai

How to master personalization with AI

Picture ecommerce in its early days: businesses were just beginning to discover the power of personalized marketing. They’d divide ...

Ciprian Borodescu

AI Product Manager | On a mission to help people succeed through the use of AI

5 best practices for nailing the ecommerce virtual assistant user experience
ai

5 best practices for nailing the ecommerce virtual assistant user experience

“Hello there, how can I help you today?”, asks the virtual shopping assistant in the lower right-hand corner ...

Vincent Caruana

Senior Digital Marketing Manager, SEO

Add InstantSearch and Autocomplete to your search experience in just 5 minutes
product

Add InstantSearch and Autocomplete to your search experience in just 5 minutes

A good starting point for building a comprehensive search experience is a straightforward app template. When crafting your application’s ...

Imogen Lovera

Senior Product Manager

Best practices of conversion-focused ecommerce website design
e-commerce

Best practices of conversion-focused ecommerce website design

The inviting ecommerce website template that balances bright colors with plenty of white space. The stylized fonts for the headers ...

Catherine Dee

Search and Discovery writer

Ecommerce product listing pages: what they are and how to optimize them for maximum conversion
e-commerce

Ecommerce product listing pages: what they are and how to optimize them for maximum conversion

Imagine an online shopping experience designed to reflect your unique consumer needs and preferences — a digital world shaped completely around ...

Vincent Caruana

Senior Digital Marketing Manager, SEO

Looking for something?

facebookfacebooklinkedinlinkedintwittertwittermailmail

Great search should guide users to their needs as quickly as possible, but achieving the right level of relevance for your site is not always straightforward and takes time to get right. With a solid search-as-a-service provider, your default ranking should already be relatively robust. Still, since every business is unique and search changes can impact KPIs and the user experience, you’ll likely want to measure the effect of adding custom ranking factors, new attributes, and more on your overall search relevance. 

With A/B testing, you can leverage data to validate your decisions in real time and make changes to your search with confidence. 

 

What is A/B testing?

A/B testing is the process of concurrently running two different versions of a user interface component while collecting metrics to determine the impact on KPIs. For a search tool, this can be useful in testing changes to relevance algorithms and configurations as there are clear metrics of success to monitor, such as click-through rate and conversions.

There are a few steps to running an A/B test for search relevance:

  1. Pick which metrics and statistical methods are relevant to the test and its outcomes.
  2. Decide the relevant questions to answer, KPIs and metrics to measure and how they relate to your proposed change. This may include a hypothesis that you’re attempting to validate.
  3. Run the two versions of your user interface concurrently to control for variability. The users should be randomly assigned to minimize any statistical noise or bias. Make sure you run the test long enough to collect statistically significant data.
  4. Analyze the test results to determine if your hypothesis was correct and/or how the changes impacted your KPIs. Did the metrics materially increase, did they decline, or did they stay relatively constant? How does this compare to historical data?
  5. Take action! If test results indicate improved KPIs, then roll out the change to all your users. If it was a failure, scrap it and iterate.

 

Why should you test your search?

Search is a fundamental part of the online experience. In fact, Forrester research found that 43% of users on retail websites go directly to the search bar. And, thanks to Google and other global search engines, users expect fast and high-quality results pages on every site they visit. A search experience optimized through A/B testing promotes user engagement by helping them quickly find what they’re looking for and ultimately convert. 

 

A/B testing is a great way to optimize your search experience right from the search box:

  • Perfect relevancy with user input. The purpose of optimizing search and tuning relevancy is to serve users better results. With A/B search testing, your users are directly involved in this process. You can track how users are interacting with your search and make improvements and updates based on real feedback and not just guesswork. Fashion platform Videdressing regularly uses A/B testing to validate product changes for customers shopping from a catalog of over 1 million products.
  • Take advantage of data-driven insights. Users are hard to predict, and A/B test results can help to show exactly how they respond to changes. When you collect enough data for an isolated change against a comparable control group, you can have confidence through statistics that the results are meaningful and make actionable decisions from them.
  • Gain knowledge without risk. A/B search testing allows you to test changes on a representative sample of your users. Since the majority of your users are unaffected, you can be confiderne that revenue and KPIs will not be affected either. 

 

5 ways to A/B test your search

Given that search applies to pretty much all industries, there are a number of ways to experiment with changes to the relevance rules and formulas to see how they impact your KPIs. Here are five important examples of types of tests you can run to improve these metrics:

 

1. See how social proof data affects your search

Leveraging social proof on your site can be highly valuable to your users and an important data point for the business. Showing likes, reviews, and comments on the site can improve visitor’s interest, confidence, and engagement with your offerings. It can also indicate to the business which products or content are particularly popular, helpful, meaningful, or in demand. Therefore, it’s often worth experimenting with ordering of the search results page  by engagement level. 

If the hypothesis is that content with more likes or shares is more relevant to users, then you could show one user segment results based on your existing sort behavior and another user segment results sorted by engagement. Track how this impacts KPIs like click-through rate to determine if the search results are in fact more relevant to users.

 

2. Understand how new searchable attributes affect your search relevance

Indexed content often changes over time as you modify the structure of your documents. If, for example, you add an additional attribute such as a “short product description,” then this content could impact how users explore and engage with it. 

Therefore, you can experiment with including this content in search terms and queries and vary how much the field is boosted relative to other fields. If boosting the new attribute shows improvements in KPIs vs. the existing search algorithm, then it is likely that it is contributing to better relevance of the search results.

 

3. Compare how a query with and without merchandising performs

Search merchandising is a powerful mechanism for promoting products that are important to your business strategies. However, promoting specific products will impact the relevance of search results.

Therefore, before rolling-out a merchandising campaign across a website, it can be valuable to A/B test the change against a control to first understand how it affects KPIs. For example, you might A/B test a merchandising technique such as boosting a category when a user types the category name” to see how this performs. If boosting categories negatively impacts the click-through rate, then you may decide that it is not worth the benefit to roll-out the campaign to the rest of your users. 

 

4. Compare search with and without personalization

Personalization allows you to customize search results to a user based on their historical data. This may include past searches, products added to carts, purchases, or other relevant metrics. As every user has their own unique needs and preferences, this can be valuable for getting them to relevant content faster. However, there are a number of ways to layer on personalization factors and thus it’s important to A/B test the changes to see how they impact KPIs. 

For example, you may hypothesize that showing users products that are in categories related to past purchases will increase their propensity to buy. By A/B testing this theory, you can have confidence that the improvement does in fact increase conversions relative to your existing algorithms—or you can try a different method of personalization instead. 

Decathlon Singapore used A/B testing to gauge the effect of personalization on each search terms and queries. Their tests showed a 50% increase in conversion rate for personalized queries, which enabled them to find the right personalisation balance for different attributes.

 

5. Add and remove some criteria from your ranking formula

There is a fine balance between having too many or too few criteria for your search rankings. Having a lot of parameters may seem like a good idea since you can highly tune the results to your customers’ needs, yet this could lead to having an overly narrow scope and reduce your flexibility to handle a diverse customer base. Conversely, a broad and simplistic ranking formula may be flexible in its scope, yet it could be too general to provide relevant results to your users. There’s no one-size-fits-all approach to optimizing this balance, so A/B testing should be used to determine how adding or removing these criteria affects the relevancy for all of your users.

 

Getting started setting up an A/B test for your search

A/B testing search is important for all websites that have a catalog of content that their users regularly browse and explore. The more relevant the results are that your users see, the more likely they are to stay engaged and continue to use your service. To do so, you’ll need a search-as-a-service provider that provides all of the tools necessary to run effective A/B tests. 

Watch our master class to see how Algolia can help you run a robust and successful A/B test for your search.

 

About the author
Louise Vollaire

Product Marketing Manager

linkedin

Is your site performing as well as it could?

Assess your search experience in less than 5 minutes.

Evaluate your search
Is your site performing as well as it could?

Recommended Articles

Powered byAlgolia Algolia Recommend

How to improve site search with Algolia A/B testing
e-commerce

Loise Mercier

The (almost) ultimate guide to site search
product

Ivana Ivanovic

Senior Content Strategist

Optimizing search performance with A/B testing
product

Nicolas Baissas