A/B test implementation checklist

Use this checklist to find implementation issues that can affect your A/B test results.

Check your events

Check that your click and conversion events are valid and usable for A/B testing. For more information, see Validate your events. If you use Revenue Analytics, send conversion events with eventSubtype set to purchase for purchases or addToCart when users add items to their cart. For revenue metrics, include the price and quantity of each purchased item and the currency.

This feature isn’t available on every plan. Refer to your pricing plan to see if it’s included.

Identify your users

Assign each user a unique, stable userToken. Use the same token for their searches and click and conversion events. Send the token with the userToken search parameter or the X-Algolia-UserToken request header. Algolia uses this token to assign users to A/B test variants. Keep it consistent across search sessions so users stay in the same variant. If you don’t provide a user token, Algolia uses the request’s IP address. Create a user token in your app instead of relying on IP addresses, which users can share. Don’t include personally identifiable information in user tokens.

Identify users in backend searches

If you use backend search, include each user’s userToken in the requests you forward to Algolia. Without a user token or a forwarded user IP address, Algolia treats searches from the same server IP address as searches from one user. If you can’t provide a user token, forward the user’s IP address with the X-Forwarded-For header.

Check your Personalization implementation

If you use Personalization, check your setup on the Personalization implementation help page in the Algolia dashboard. Send the same userToken with searches and click and conversion events.

This feature isn’t available on every plan. Refer to your pricing plan to see if it’s included.

Handle anonymous users

If each anonymous user has a unique token that stays the same across search sessions, include them in your A/B test. Don’t exclude users solely because they aren’t signed in. If users share a token, Algolia treats them as one user. If you can’t provide a unique, stable token, exclude those searches from A/B testing by setting enableABTest to false. The following example checks for a missing or anonymous user token (null, undefined, or YOUR_ANONYMOUS_USER_TOKEN). It sets enableABTest to false for those search requests.

namespace Algolia;

using System;
using System.Collections.Generic;
using System.Net.Http;
using System.Text.Json;
using Algolia.Search.Clients;
using Algolia.Search.Http;
using Algolia.Search.Models.Search;

class AbTestImplementationChecklist
{
  private static string GetUserToken()
  {
    // Implement your logic here
    return "";
  }

  async Task Main(string[] args)
  {
    var client = new SearchClient(new SearchConfig("ALGOLIA_APPLICATION_ID", "ALGOLIA_API_KEY"));

    // Set the query and get the current user token
    var query = "User search query";
    var userToken = GetUserToken();

    // Set the searchParams
    var searchParamsObject = new SearchParamsObject { Query = query };

    // Is the user token anonymous?
    if (string.IsNullOrEmpty(userToken) || userToken == "YOUR_ANONYMOUS_USER_TOKEN")
    {
      // Disable A/B testing for this request
      searchParamsObject.EnableABTest = false;
    }
    else
    {
      // Set the user token to the current user token
      searchParamsObject.UserToken = userToken;
    }

    var searchParams = new SearchParams(searchParamsObject);

    try
    {
      // Perform the searchSingleIndex
      var result = await client.SearchSingleIndexAsync<Hit>("INDEX_NAME", searchParams);
      // SearchSingleIndex results
      Console.WriteLine(result);
    }
    catch (Exception err)
    {
      // SearchSingleIndex errors
      Console.Error.WriteLine(err);
    }
  }
}

import 'package:algolia_client_search/algolia_client_search.dart';

String getUserToken() {
  // Implement your logic here
  return "";
}

void abTestImplementationChecklist() async {
  final client =
      SearchClient(appId: 'ALGOLIA_APPLICATION_ID', apiKey: 'ALGOLIA_API_KEY');

  // Set the query and get the current user token
  final query = "User search query";
  final userToken = getUserToken();

  // Set the searchParams
  final SearchParamsObject searchParams;

  // Is the user token anonymous?
  if (userToken.isEmpty || userToken == "YOUR_ANONYMOUS_USER_TOKEN") {
    // Disable A/B testing for this request
    searchParams = SearchParamsObject(
      query: query,
      enableABTest: false,
    );
  } else {
    // Set the user token to the current user token
    searchParams = SearchParamsObject(
      query: query,
      userToken: userToken,
    );
  }

  try {
    // Perform the searchSingleIndex
    final result = await client.searchSingleIndex(
      indexName: "INDEX_NAME",
      searchParams: searchParams,
    );
    // SearchSingleIndex results
    print(result);
  } catch (err) {
    // SearchSingleIndex errors
    print(err);
  }
}

package main

import (
	"github.com/algolia/algoliasearch-client-go/v4/algolia/search"
)

func getUserToken() (string, error) {
	// Implement your logic here
	return "", nil
}

func abTestImplementationChecklist() {
	client, err := search.NewClient("ALGOLIA_APPLICATION_ID", "ALGOLIA_API_KEY")
	if err != nil {
		// The client can fail to initialize if you pass an invalid parameter.
		panic(err)
	}

	// Set the query and get the current user token
	query := "User search query"

	userToken, err := getUserToken()
	if err != nil {
		panic(err)
	}

	// Set the searchParams
	var searchParams *search.SearchParams

	// Is the user token anonymous?
	if userToken == "" || userToken == "YOUR_ANONYMOUS_USER_TOKEN" {
		// Disable A/B testing for this request
		searchParams = search.SearchParamsObjectAsSearchParams(
			search.NewSearchParamsObject().
				SetQuery(query).
				SetEnableABTest(false),
		)
	} else {
		// Set the user token to the current user token
		searchParams = search.SearchParamsObjectAsSearchParams(
			search.NewSearchParamsObject().
				SetQuery(query).
				SetUserToken(userToken),
		)
	}

	// Perform the searchSingleIndex
	_, err = client.SearchSingleIndex(client.NewApiSearchSingleIndexRequest(
		"INDEX_NAME").WithSearchParams(searchParams))
	if err != nil {
		// SearchSingleIndex errors
		panic(err)
	}
}

package com.algolia;

import com.algolia.api.SearchClient;
import com.algolia.config.*;
import com.algolia.model.search.*;

public class abTestImplementationChecklist {

  private static String getUserToken() {
    // Implement your logic here
    return "";
  }

  public static void main(String[] args) throws Exception {
    SearchClient client = new SearchClient("ALGOLIA_APPLICATION_ID", "ALGOLIA_API_KEY");

    // Set the query and get the current user token
    String query = "User search query";
    String userToken = getUserToken();

    // Set the searchParams
    SearchParamsObject searchParams = new SearchParamsObject().setQuery(query);

    // Is the user token anonymous?
    if (userToken == null || userToken.isEmpty() || userToken.equals("YOUR_ANONYMOUS_USER_TOKEN")) {
      // Disable A/B testing for this request
      searchParams.setEnableABTest(false);
    } else {
      // Set the user token to the current user token
      searchParams.setUserToken(userToken);
    }

    try {
      // Perform the searchSingleIndex
      SearchResponse result = client.searchSingleIndex("INDEX_NAME", searchParams, Hit.class);
      // SearchSingleIndex results
      System.out.println(result);
    } catch (Exception err) {
      // SearchSingleIndex errors
      System.err.println(err);
    }
  }
}

import { algoliasearch } from 'algoliasearch';

import type { SearchParams } from 'algoliasearch';

const client = algoliasearch('ALGOLIA_APPLICATION_ID', 'ALGOLIA_API_KEY');

// Implement your logic here
const getUserToken = () => {
  return '';
};

// Set the query and get the current user token
const query = 'User search query';
const userToken = getUserToken();

// Set the searchParams
const searchParams: SearchParams = {
  query,
};

// Is the user token anonymous?
if (userToken === null || userToken === undefined || userToken === 'YOUR_ANONYMOUS_USER_TOKEN') {
  // Disable A/B testing for this request
  searchParams.enableABTest = false;
} else {
  // Set the user token to the current user token
  searchParams.userToken = userToken;
}

try {
  // Perform the searchSingleIndex
  const result = await client.searchSingleIndex({ indexName: 'indexName', searchParams: searchParams });
  // SearchSingleIndex results
  console.log(result);
} catch (err) {
  // SearchSingleIndex errors
  console.error(err);
}

import com.algolia.client.api.SearchClient
import com.algolia.client.configuration.*
import com.algolia.client.extensions.*
import com.algolia.client.model.search.*
import com.algolia.client.transport.*

fun getUserToken(): String {
  // Implement your logic here
  return ""
}

suspend fun abTestImplementationChecklist() {
  val client = SearchClient(appId = "ALGOLIA_APPLICATION_ID", apiKey = "ALGOLIA_API_KEY")

  // Set the query and get the current user token
  val query = "User search query"
  val userToken = getUserToken()

  // Set the searchParams
  val searchParams: SearchParamsObject

  // Is the user token anonymous?
  if (userToken.isEmpty() || userToken == "YOUR_ANONYMOUS_USER_TOKEN") {
    // Disable A/B testing for this request
    searchParams = SearchParamsObject(query = query, enableABTest = false)
  } else {
    // Set the user token to the current user token
    searchParams = SearchParamsObject(query = query, userToken = userToken)
  }

  try {
    // Perform the searchSingleIndex
    val result =
      client.searchSingleIndex(indexName = "INDEX_NAME", searchParams = searchParams)
    // SearchSingleIndex results
    println(result)
  } catch (err: Exception) {
    // SearchSingleIndex errors
    println(err)
  }
}

<?php

require __DIR__.'/../vendor/autoload.php';
use Algolia\AlgoliaSearch\Api\SearchClient;
use Algolia\AlgoliaSearch\Model\Search\SearchParamsObject;

$getUserToken = function (): string {
    // Implement your logic here
    return '';
};

$client = SearchClient::create('ALGOLIA_APPLICATION_ID', 'ALGOLIA_API_KEY');

// Set the query and get the current user token
$query = 'User search query';
$userToken = $getUserToken();

// Set the searchParams
$searchParams = (new SearchParamsObject())
    ->setQuery($query)
;

// Is the user token anonymous?
if (null === $userToken || '' === $userToken || 'YOUR_ANONYMOUS_USER_TOKEN' === $userToken) {
    // Disable A/B testing for this request
    $searchParams->setEnableABTest(false);
} else {
    // Set the user token to the current user token
    $searchParams->setUserToken($userToken);
}

try {
    // Perform the searchSingleIndex
    $result = $client->searchSingleIndex(
        'INDEX_NAME',
        $searchParams,
    );
    // SearchSingleIndex results
    var_dump($result);
} catch (Exception $err) {
    // SearchSingleIndex errors
    error_log($err->getMessage());
}

from algoliasearch.search.client import SearchClientSync

from algoliasearch.search.models.search_params import SearchParams


def _get_user_token() -> str:
    # Implement your logic here
    return ""


_client = SearchClientSync("ALGOLIA_APPLICATION_ID", "ALGOLIA_API_KEY")


# Set the query and get the current user token
query = "User search query"
user_token = _get_user_token()

# Set the searchParams
# Is the user token anonymous?
if user_token is None or user_token == "" or user_token == "YOUR_ANONYMOUS_USER_TOKEN":
    # Disable A/B testing for this request
    search_params = SearchParams(
        query=query,
        enable_ab_test=False,
    )
else:
    # Set the user token to the current user token
    search_params = SearchParams(
        query=query,
        user_token=user_token,
    )

try:
    # Perform the searchSingleIndex
    result = _client.search_single_index(
        index_name="INDEX_NAME",
        search_params=search_params,
    )
    # SearchSingleIndex results
    print(result)
except Exception as err:
    # SearchSingleIndex errors
    print(err)

require "algolia"

def get_user_token
  # Implement your logic here
  ""
end

client = Algolia::SearchClient.create("ALGOLIA_APPLICATION_ID", "ALGOLIA_API_KEY")

# Set the query and get the current user token
query = "User search query"
user_token = get_user_token

# Set the searchParams
search_params = Algolia::Search::SearchParamsObject.new(
  query: query
)

# Is the user token anonymous?
if user_token.nil? || user_token.empty? || user_token == "YOUR_ANONYMOUS_USER_TOKEN"
  # Disable A/B testing for this request
  search_params.enable_ab_test = false
else
  # Set the user token to the current user token
  search_params.user_token = user_token
end

begin
  # Perform the searchSingleIndex
  result = client.search_single_index("INDEX_NAME", search_params)
  # SearchSingleIndex results
  puts(result)
rescue => err
  # SearchSingleIndex errors
  puts(err)
end

import scala.concurrent.Future
import scala.concurrent.ExecutionContext.Implicits.global

import algoliasearch.api.SearchClient
import algoliasearch.config.*
import algoliasearch.extension.SearchClientExtensions
import algoliasearch.search.SearchParamsObject

val getUserToken: String = {
  "" // Implement your logic here
}

def abTestImplementationChecklist(): Future[Unit] = {
  val client = SearchClient(appId = "ALGOLIA_APPLICATION_ID", apiKey = "ALGOLIA_API_KEY")

  // Set the query and get the current user token
  val query = "User search query"
  val userToken = getUserToken

  // Set the searchParams
  var searchParams = SearchParamsObject(
    query = Some(query)
  )

  // Is the user token anonymous?
  if (userToken == null || userToken.isEmpty || userToken == "YOUR_ANONYMOUS_USER_TOKEN") {
    // Disable A/B testing for this request
    searchParams = searchParams.copy(enableABTest = Some(false))
  } else {
    // Set the user token to the current user token
    searchParams = searchParams.copy(userToken = Some(userToken))
  }

  // Perform the searchSingleIndex
  client
    .searchSingleIndex(
      indexName = "INDEX_NAME",
      searchParams = Some(searchParams)
    )
    .map { result =>
      // SearchSingleIndex results
      println(result)
    }
    .recover { case err: Exception =>
      // SearchSingleIndex errors
      println(s"An error occurred: ${err.getMessage}")
    }
}

import Foundation
#if os(Linux) // For linux interop
    import FoundationNetworking
#endif

import AlgoliaCore
import AlgoliaSearch

let getUserToken = { "" } // Implement your logic here

func abTestImplementationChecklist() async throws {
    let client = try SearchClient(appID: "ALGOLIA_APPLICATION_ID", apiKey: "ALGOLIA_API_KEY")

    // Set the query and get the current user token
    let query = "User search query"
    let userToken = getUserToken()

    // Set the searchParams
    // Is the user token anonymous?
    let searchParams =
        if userToken.isEmpty || userToken == "YOUR_ANONYMOUS_USER_TOKEN" {
            // Disable A/B testing for this request
            SearchSearchParams.searchSearchParamsObject(
                SearchSearchParamsObject(query: query, enableABTest: false)
            )
        } else {
            // Set the user token to the current user token
            SearchSearchParams.searchSearchParamsObject(
                SearchSearchParamsObject(query: query, userToken: userToken)
            )
        }

    do {
        // Perform the searchSingleIndex
        let result: SearchResponse<Hit> = try await client.searchSingleIndex(
            indexName: "INDEX_NAME",
            searchParams: searchParams
        )
        // SearchSingleIndex results
        print(result)
    } catch {
        // SearchSingleIndex errors
        print(error)
    }
}

Exclude internal searches

Exclude searches from internal dashboards, testing tools, and administrative pages by setting enableABTest to false.

Avoid overrides of A/B test settings

Don’t send search parameters that override the settings you’re comparing in an A/B test. For example, to compare results with and without Personalization, turn it off for variant A and on for variant B. Don’t send enablePersonalization: true with every search request. Otherwise, both variants can use Personalization, so the test doesn’t measure the effect of turning it on.

Manage bot traffic

Bot-generated searches can distort and other test metrics. Although Algolia excludes outlier users when calculating A/B test results, this filtering doesn’t necessarily block bot searches. To reduce bot traffic:

Set rate limits. Configure API key rate limits to restrict search requests.
Restrict HTTP referrers. Allow only expected referrers for your API keys. Don’t rely on this restriction alone to block bots because clients can spoof the Referer header.
Configure robots.txt. Disallow crawling of your search pages. These rules apply only to bots that respect robots.txt.

Search and Discovery platform

Optimization and Personalization

AI-powered experiences

Production and scale

A/B test implementation checklist

Check your events

Identify your users

Identify users in backend searches

Check your Personalization implementation

Handle anonymous users

Exclude internal searches

Avoid overrides of A/B test settings

Manage bot traffic

​Check your events

​Identify your users

​Identify users in backend searches

​Check your Personalization implementation

​Handle anonymous users

​Exclude internal searches

​Avoid overrides of A/B test settings

​Manage bot traffic

Check your events

Identify your users

Identify users in backend searches

Check your Personalization implementation

Handle anonymous users

Exclude internal searches

Avoid overrides of A/B test settings

Manage bot traffic