5 ways to A/B test your search for relevance

Great search should guide users to their needs as quickly as possible, but achieving the right level of relevance for your site is not always straightforward and takes time to get right. With a solid search-as-a-service provider, your default ranking should already be relatively robust. Still, since every business is unique and search changes can impact KPIs and the user experience, you’ll likely want to measure the effect of adding custom ranking factors, new attributes, and more on your overall search relevance. 

With A/B testing, you can leverage data to validate your decisions in real time and make changes to your search with confidence. 

What is A/B testing?

A/B testing is the process of concurrently running two different versions of a user interface component while collecting metrics to determine the impact on KPIs. For a search tool, this can be useful in testing changes to relevance algorithms and configurations as there are clear metrics of success to monitor, such as click-through rate and conversions.

There are a few steps to running an A/B test for search relevance:

1. Pick which metrics and statistical methods are relevant to the test and its outcomes.
2. Decide the relevant questions to answer, KPIs and metrics to measure and how they relate to your proposed change. This may include a hypothesis that you’re attempting to validate.
3. Run the two versions of your user interface concurrently to control for variability. The users should be randomly assigned to minimize any statistical noise or bias. Make sure you run the test long enough to collect statistically significant data.
4. Analyze the test results to determine if your hypothesis was correct and/or how the changes impacted your KPIs. Did the metrics materially increase, did they decline, or did they stay relatively constant? How does this compare to historical data?
5. Take action! If test results indicate improved KPIs, then roll out the change to all your users. If it was a failure, scrap it and iterate.

Why should you test your search?

Search is a fundamental part of the online experience. In fact, Forrester research found that 43% of users on retail websites go directly to the search bar. And, thanks to Google and other global search engines, users expect fast and high-quality results pages on every site they visit. A search experience optimized through A/B testing promotes user engagement by helping them quickly find what they’re looking for and ultimately convert. 

A/B testing is a great way to optimize your search experience right from the search box:

• Perfect relevancy with user input. The purpose of optimizing search and tuning relevancy is to serve users better results. With A/B search testing, your users are directly involved in this process. You can track how users are interacting with your search and make improvements and updates based on real feedback and not just guesswork. Fashion platform Videdressing regularly uses A/B testing to validate product changes for customers shopping from a catalog of over 1 million products.
• Take advantage of data-driven insights. Users are hard to predict, and A/B test results can help to show exactly how they respond to changes. When you collect enough data for an isolated change against a comparable control group, you can have confidence through statistics that the results are meaningful and make actionable decisions from them.
• Gain knowledge without risk. A/B search testing allows you to test changes on a representative sample of your users. Since the majority of your users are unaffected, you can be confident that revenue and KPIs will not be affected either. 

5 ways to A/B test your search

Given that search applies to pretty much all industries, there are a number of ways to experiment with changes to the relevance rules and formulas to see how they impact your KPIs. Here are five important examples of types of tests you can run to improve these metrics:

1. See how social proof data affects your search

Leveraging social proof on your site can be highly valuable to your users and an important data point for the business. Showing likes, reviews, and comments on the site can improve visitor’s interest, confidence, and engagement with your offerings. It can also indicate to the business which products or content are particularly popular, helpful, meaningful, or in demand. Therefore, it’s often worth experimenting with ordering of the search results page  by engagement level. 

If the hypothesis is that content with more likes or shares is more relevant to users, then you could show one user segment results based on your existing sort behavior and another user segment results sorted by engagement. Track how this impacts KPIs like click-through rate to determine if the search results are in fact more relevant to users.

2. Understand how new searchable attributes affect your search relevance

Indexed content often changes over time as you modify the structure of your documents. If, for example, you add an additional attribute such as a “short product description,” then this content could impact how users explore and engage with it. 

Therefore, you can experiment with including this content in search terms and queries and vary how much the field is boosted relative to other fields. If boosting the new attribute shows improvements in KPIs vs. the existing search algorithm, then it is likely that it is contributing to better relevance of the search results.

3. Compare how a query with and without merchandising performs

Search merchandising is a powerful mechanism for promoting products that are important to your business strategies. However, promoting specific products will impact the relevance of search results.

Therefore, before rolling-out a merchandising campaign across a website, it can be valuable to A/B test the change against a control to first understand how it affects KPIs. For example, you might A/B test a merchandising technique such as boosting a category when a user types the category name” to see how this performs. If boosting categories negatively impacts the click-through rate, then you may decide that it is not worth the benefit to roll-out the campaign to the rest of your users. 

4. Compare search with and without personalization

Personalization allows you to customize search results to a user based on their historical data. This may include past searches, products added to carts, purchases, or other relevant metrics. As every user has their own unique needs and preferences, this can be valuable for getting them to relevant content faster. However, there are a number of ways to layer on personalization factors and thus it’s important to A/B test the changes to see how they impact KPIs. 

For example, you may hypothesize that showing users products that are in categories related to past purchases will increase their propensity to buy. By A/B testing this theory, you can have confidence that the improvement does in fact increase conversions relative to your existing algorithms—or you can try a different method of personalization instead. 

Decathlon Singapore used A/B testing to gauge the effect of personalization on each search terms and queries. Their tests showed a 50% increase in conversion rate for personalized queries, which enabled them to find the right personalization balance for different attributes.

5. Add and remove some criteria from your ranking formula

There is a fine balance between having too many or too few criteria for your search rankings. Having a lot of parameters may seem like a good idea since you can highly tune the results to your customers’ needs, yet this could lead to having an overly narrow scope and reduce your flexibility to handle a diverse customer base. Conversely, a broad and simplistic ranking formula may be flexible in its scope, yet it could be too general to provide relevant results to your users. There’s no one-size-fits-all approach to optimizing this balance, so A/B testing should be used to determine how adding or removing these criteria affects the relevancy for all of your users.

Getting started setting up an A/B test for your search

A/B testing search is important for all websites that have a catalog of content that their users regularly browse and explore. The more relevant the results are that your users see, the more likely they are to stay engaged and continue to use your service. To do so, you’ll need a search-as-a-service provider that provides all of the tools necessary to run effective A/B tests. 

Watch our master class to see how Algolia can help you run a robust and successful A/B test for your search.