A/B test implementation checklist
On this page
Use this checklist to avoid common issues with statistical significance and unexpected test results.
Check your events
Check that all events are in the right format and are valid. For more information, see Validate your events.
Identify your users
Link click and conversion events with user profiles by assigning a unique identifier, userToken
, to each user.
Handle backend search users
If you use backend search, since all searches appear to come from your server’s IP address, Algolia treats them as coming from a single user.
To differentiate between users, include the userToken
or the user’s IP address when forwarding queries from your server to Algolia.
Check Personalization implementation
If you’re using Personalization, check it’s working as expected with the Personalization implementation help page on the Algolia dashboard. Your queries should have associated user tokens.
Personalization is available on the Build and Premium pricing plans.
Handle anonymous users
Sometimes, you may have a mix of anonymous and identified users. For example, you may have assigned an anonymous token to users who haven’t yet accepted cookie consent. Since anonymous users share the same user token, Algolia treats them as coming from a single user.
To avoid skewing results, turn off A/B testing for queries from anonymous users.
For example, the following checks for an anonymous user token (null
, undefined
, or YOUR_ANONYMOUS_USER_TOKEN
) and appends the { "[enableABTest](parameter_url('enableABTest'))": false }
parameter to the outgoing query.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
// Set the query and get the current user token
const query = 'User search query';
const userToken = getUserToken();
// Clear the request options
let requestOptions = {};
// Is the user token anonymous?
if (userToken === null || userToken === undefined || userToken === 'YOUR_ANONYMOUS_USER_TOKEN') {
// Disable A/B testing for this request
requestOptions.enableABTest = false;
} else {
// Set the user token to the current user token
requestOptions.userToken = getUserToken();
}
// Perform the search
index.search(query, requestOptions)
.then(result => {
// Search results
console.log(result);
})
.catch(error => {
// Search errors
console.error(error);
});
Exclude internal searches
Exclude non-user searches (such as from a dashboard or internal page) from A/B testing by setting enableABTest
to false.
Avoid overriding A/B test settings with query parameters
For accurate results, don’t change any query parameters you initially set in the A/B test. For example, you turn on Personalization for variant B and all queries. This wouldn’t give meaningful results because all users see the personalized experience, making it difficult to tell if variant B has any impact.
Manage bot traffic
Bots crawling your web pages and performing searches will generate outlier traffic. Although outliers are automatically removed from the A/B test results, you should still take steps to avoid them because they can affect the click-through rate.
To control and restrict bot activity and avoid skewing results:
- Set rate limits. Set API keys with rate limits to control the number of API calls per hour and IP address.
- Use HTTP referrers. However, don’t rely solely on referrer URLs sent through the
referer
orOrigin HTTP
headers to secure your data. Malicious actors can tamper with them, compromising your security, and some browsers, like Brave, don’t send these headers. - Update robots.txt. Configure
robots.txt
to prevent bots from accessing your search pages.