Optimizing search performance with A/B testing

Nicolas Baissas

To date, businesses have been testing search relevance in the dark—with little to no performance data or post-query metrics to inform optimizing search results for engagement. Algolia’s Analytics API and Click Analytics solutions started solving this challenge by providing you a complete view into the entire search lifecycle—from query to click-through to conversion.

The A/B testing feature rolled out today on our Enterprise plans empowers you to test performance impact of a change in the configuration of your search settings. The best part: you can create A/B tests entirely from the dashboard, without a single line of code (and of course, it is also doable entirely from the API).

Why A/B testing?

Search and discovery are core to the digital experience, and relevance is the one aspect of search that can’t be overlooked. Nothing will have as much impact on the quality of your users’ experience than the relevance of results returned by your search engine.

Let’s take e-commerce as an example.

Optimize for conversions: there are numerous criteria an e-commerce website could use to improve relevance so that users can easily find what they are looking for and convert faster; for example, promoting product sales, popularity, ratings, in search results. But it can quickly get overwhelming. Where to start? How to make sure those new criteria aren’t counterproductive? A/B Testing allows continuous, iterative optimization, criterion by criterion.
Validate business decisions: there are various parameters that define an e-commerce business. Maybe your operational costs require you to favorise high-margin products. But how to make sure doing so does not have negative impact on sales? A/B Testing adds confidence to the decision making process.

From the parsing of textual tokens, to the impact of your business metrics on the ranking formula, to the importance of the proximity of words in the query, Algolia offers dozens of features and settings allowing you to fine-tune the relevance to achieve great search results.

For Algolia users, here is how

Let’s take an example where you have an index currently used in production. Let’s call it indexA.

The ranking strategy on this index currently relies on the number_of_views of each result. You’d like to know if changing the ranking strategy to rely on a number_of_likes instead of the number_of_views would lead to better search conversions.

A/B testing allows you to do exactly that, by following these steps:

Create a replica of the indexA—we’ll call it indexB (make sure beforehand that you have enough space to duplicate the data)
Change the c ustom ranking of indexB to use number_of_likes instead of number_of_views
If you haven’t done it yet, implement cl ick analytics. It will be used to measure performance improvements caused by your configuration change in index B
Go to the A/B testing feature on the dashboard, and enable an A/B test between indexA and indexB
Wait to get enough traffic until the significance score of the AB test reaches 95%, and that’s it!

Depending on the result, you’ll know if your new ranking strategy leads to more clicks and conversions on your search, so you can proceed with the changes with confidence!

We hope this feature will help you iterate faster on your configuration and achieve the best relevance for your search. And we certainly plan to guide you along the way and help you make the most of it!

You can always read the documentation or visit the product page to learn more.

Don’t hesitate to get in touch and share your questions and feedback: shoot us an email, tweet to us, or submit a contact form.