Is your site performing as well as it could?
Assess your search experience in less than 5 minutes.
Sorry, there is no results for this query
Great search should guide users to their needs as quickly as possible, but achieving the right level of relevance for your site is not always straightforward and takes time to get right. With a solid search-as-a-service provider, your default ranking should already be relatively robust. Still, since every business is unique and search changes can impact KPIs and the user experience, you’ll likely want to measure the effect of adding custom ranking factors, new attributes, and more on your overall search relevance.
With A/B testing, you can leverage data to validate your decisions in real time and make changes to your search with confidence.
A/B testing is the process of concurrently running two different versions of a user interface component while collecting metrics to determine the impact on KPIs. For a search tool, this can be useful in testing changes to relevance algorithms and configurations as there are clear metrics of success to monitor, such as click-through rate and conversions.
There are a few steps to running an A/B test for search relevance:
Search is a fundamental part of the online experience. In fact, Forrester research found that 43% of users on retail websites go directly to the search bar. And, thanks to Google and other global search engines, users expect fast and high-quality results pages on every site they visit. A search experience optimized through A/B testing promotes user engagement by helping them quickly find what they’re looking for and ultimately convert.
A/B testing is a great way to optimize your search experience right from the search box:
Given that search applies to pretty much all industries, there are a number of ways to experiment with changes to the relevance rules and formulas to see how they impact your KPIs. Here are five important examples of types of tests you can run to improve these metrics:
Leveraging social proof on your site can be highly valuable to your users and an important data point for the business. Showing likes, reviews, and comments on the site can improve visitor’s interest, confidence, and engagement with your offerings. It can also indicate to the business which products or content are particularly popular, helpful, meaningful, or in demand. Therefore, it’s often worth experimenting with ordering of the search results page by engagement level.
If the hypothesis is that content with more likes or shares is more relevant to users, then you could show one user segment results based on your existing sort behavior and another user segment results sorted by engagement. Track how this impacts KPIs like click-through rate to determine if the search results are in fact more relevant to users.
Indexed content often changes over time as you modify the structure of your documents. If, for example, you add an additional attribute such as a “short product description,” then this content could impact how users explore and engage with it.
Therefore, you can experiment with including this content in search terms and queries and vary how much the field is boosted relative to other fields. If boosting the new attribute shows improvements in KPIs vs. the existing search algorithm, then it is likely that it is contributing to better relevance of the search results.
Search merchandising is a powerful mechanism for promoting products that are important to your business strategies. However, promoting specific products will impact the relevance of search results.
Therefore, before rolling-out a merchandising campaign across a website, it can be valuable to A/B test the change against a control to first understand how it affects KPIs. For example, you might A/B test a merchandising technique such as boosting a category when a user types the category name” to see how this performs. If boosting categories negatively impacts the click-through rate, then you may decide that it is not worth the benefit to roll-out the campaign to the rest of your users.
Personalization allows you to customize search results to a user based on their historical data. This may include past searches, products added to carts, purchases, or other relevant metrics. As every user has their own unique needs and preferences, this can be valuable for getting them to relevant content faster. However, there are a number of ways to layer on personalization factors and thus it’s important to A/B test the changes to see how they impact KPIs.
For example, you may hypothesize that showing users products that are in categories related to past purchases will increase their propensity to buy. By A/B testing this theory, you can have confidence that the improvement does in fact increase conversions relative to your existing algorithms—or you can try a different method of personalization instead.
Decathlon Singapore used A/B testing to gauge the effect of personalization on each search terms and queries. Their tests showed a 50% increase in conversion rate for personalized queries, which enabled them to find the right personalisation balance for different attributes.
There is a fine balance between having too many or too few criteria for your search rankings. Having a lot of parameters may seem like a good idea since you can highly tune the results to your customers’ needs, yet this could lead to having an overly narrow scope and reduce your flexibility to handle a diverse customer base. Conversely, a broad and simplistic ranking formula may be flexible in its scope, yet it could be too general to provide relevant results to your users. There’s no one-size-fits-all approach to optimizing this balance, so A/B testing should be used to determine how adding or removing these criteria affects the relevancy for all of your users.
A/B testing search is important for all websites that have a catalog of content that their users regularly browse and explore. The more relevant the results are that your users see, the more likely they are to stay engaged and continue to use your service. To do so, you’ll need a search-as-a-service provider that provides all of the tools necessary to run effective A/B tests.
Watch our master class to see how Algolia can help you run a robust and successful A/B test for your search.