API Reference / REST API / A/B test REST API
Sep. 23, 2019

A/B test REST API

Overview

A/B testing enables seamless relevance tuning experimentation by measuring the impact they have on the end users.

Note: each endpoint targets a single index.

Domain

The Analytics API and A/B test API can be reached from multiple domains, each specific to a region. You should use the domain that matches the region where your analytics data is stored and processed. View your analytics region.

The following domains are available:

Note: https://analytics.algolia.com is an alias of https://analytics.us.algolia.com.

Common Parameters

Every endpoint accepts the following set of parameters:

  • X-Algolia-Application-Id: The appID. Can also be passed through HTTP headers.
  • X-Algolia-API-Key: Auth token. Can also be passed through HTTP headers.

ACL

For adding, stopping, and deleting A/B tests, the X-Algolia-API-Key needs to have the editSettings ACL.

For getting and listing A/B tests, X-Algolia-API-Key needs to have the analytics ACL.

Notes on usage

Some important considerations to keep in mind.

  • You will need to have both analytics and clickAnalytics enabled on your account.
  • Currently the feature is limited to 2 concurrent variants (A and B). We cannot target the same index in the 2 variants.
  • The engine will spread traffic for both variants depending on the A/B test configuration, so there is no modification to perform on the frontend.
  • The ordering in which variants are defined is important; the first variant index will be the “control index” meaning that when targeting this index, some traffic will be automatically redirected to the other variant index (how often is determined by the allotment percentage). However, when targeting the second variant index, no redirect will happen.
  • While running a test, the Analytics Dashboard will show the combined metrics for both indices (cf previous point). However, this doesn’t fully make sense for some metrics (like no results). We might iterate on this in the future.
  • We do not provide a way to validate that the UX will work with both variants for now so users should handle this on their side. We therefore recommend running your B index in your current UI before starting the A/B test
  • A/B test analytics will not provide a daily breakdown but just a global view. A daily breakdown is actually not very useful when comparing the variants.
  • The full A/B test history will be kept until the app is deleted or a test is explicitly deleted.
  • The ABTest status might be:
    • active: the Analytics created the A/B test and performed a successful request to the engine
    • stopped: the A/B test was stopped by a user: it was deleted from the engine but we have to keep the data for historical purposes
    • expired: the A/B test reached its end date and was automatically stopped. It is removed from the engine but the metadata/metrics are kept
    • failed: the A/B test creation failed

Validation Error

In case of error, the API will return a payload in the following format:

1
2
3
4
{
  “status”: 123,
  “message”:”The error message”
}

A/B test Endpoints

Quick Reference

Verb Path Method

POST

/2/abtests

Add A/B test

GET

/2/abtests/{id}

Get A/B test

GET

/2/abtests

List A/B tests

POST

/2/abtests/{id}/stop

Stop A/B test

DELETE

/2/abtests/{id}

Delete A/B test

Add A/B test

Path: /2/abtests
HTTP Verb: POST
Required API Key: any key with the setSettings ACL

Description:
Creates a new A/B test with provided configuration.

You can set an A/B test on two different indices with different settings, or on the same index with different search parameters by providing a customSearchParameters setting on one of the variants.

Parameters:

name
type: string
Required

A/B test name.

variants
type: object
default: 20
Required

List of 2 variants for the A/B test.

All properties are required, except description.

1
2
3
4
5
6
7
8
9
10
11
12
[
  {
    "index": string // mandatory
    "trafficPercentage": int // mandatory
    "description": string // optional
  },
  {
    "index": string // mandatory
    "trafficPercentage": int // mandatory
    "description": string // optional
  }
]

To create an A/B test for search parameters only, you must provide customSearchParameters for one of the variants:

1
2
3
4
5
6
7
8
9
10
11
12
13
[
  {
    "index": string // mandatory
    "trafficPercentage": int // mandatory
    "description": string // optional
  },
  {
    "index": string // mandatory
    "trafficPercentage": int // mandatory
    "customSearchParameters": object // mandatory for one of the variants
    "description": string // optional
  }
]
endAt
type: Date, UTC. Inclusive.
default: Today, 23:59:59pm, UTC
Optional

End date for the A/B test expressed as YYYY-MM-DDThh:mm:ssZ.

Errors:

  • 400: Invalid signature or invalid request
  • 401: AppID or API Key missing
  • 402: Feature not enabled
  • 403: Invalid credentials
  • 404: index not found
  • 422: Invalid payload, A/B test already running on one of the indices
  • 500: Internal error (possible e.g. if insertion query fails or engine query fails)

Example:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
curl -X POST \
     -H "X-Algolia-API-Key: ${API_KEY}" \
     -H "X-Algolia-Application-Id: ${APPLICATION_ID}" \
     --data-binary '{
      "name": "Custom Ranking salesRank Test",
      "variants": [
        {
          "index": "atis-abtest-default",
          "description": "current production index",
          "trafficPercentage": 70
        },
        {
          "index": "atis-abtest-salesRank",
          "trafficPercentage": 30
        }
      ],
      "endAt": "2018-05-17T23:59:59Z"
    }' \
    "https://analytics.algolia.com/2/abtests"

Upon success, the response is 200 OK.

1
2
3
4
5
{
  "abTestID":78,
  "taskID":111887230,
  "index":"atis-abtest-default"
}

Create an A/B test with the second variant enabling personalization:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
curl -X POST \
     -H "X-Algolia-API-Key: ${API_KEY}" \
     -H "X-Algolia-Application-Id: ${APPLICATION_ID}" \
     --data-binary '{
      "name": "Custom Ranking salesRank Test",
      "variants": [
        {
          "index": "atis",
          "trafficPercentage": 70
        },
        {
          "index": "atis",
          "trafficPercentage": 30,
          "customSearchParameters": {
              "enablePersonalization": true
          }
        }
      ],
      "endAt": "2018-05-17T23:59:59Z"
    }' \
    "https://analytics.algolia.com/2/abtests"

Get A/B test

Path: /2/abtests/{id}
HTTP Verb: GET
Required API Key: any key with the analytics ACL

Description:
Returns metadata and metrics for A/B test id.

Behaves in the same way as GET /2/abtests however the endpoint will return 403.

Parameters:

id
type: int
Required

A/B test ID.

Response

clickSignificance
type: float

A/B test significance based on click data. Should be > 0.95 to be considered significant (no matter which variant is winning).

conversionSignificance
type: float

A/B test significance based on conversion data. Should be > 0.95 to be considered significant (no matter which variant is winning)

averageClickPosition
type: int

Average click position for the variant.

clickCount
type: int

Distinct click count for the variant.

clickThroughRate
type: float

Click through rate for the variant.

conversionCount
type: int

Distinct conversion count for the variant.

conversionRate
type: float

Conversion rate for the variant.

customSearchParameters
type: object

Search parameters applied to the variant.

Errors:

  • 401: AppID or API Key missing
  • 403: Forbidden in case the API Key has no access to the main index for this A/B test.

Example:

1
2
3
4
curl -X GET \
     -H "X-Algolia-API-Key: ${API_KEY}" \
     -H "X-Algolia-Application-Id: ${APPLICATION_ID}" \
    "https://analytics.algolia.com/2/abtests/${AB_TEST_ID}"

Upon success, the response is 200 OK.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
{
  "abTestID": 78,
  "clickSignificance": 1,
  "conversionSignificance": 0.9937,
  "createdAt": "2018-05-15T17:52:15.644906Z",
  "endAt": "2018-05-17T23:59:59Z",
  "name": "Custom Ranking salesRank Test",
  "status": "stopped",
  "variants": [
    {
      "averageClickPosition": 0,
      "clickCount": 17115,
      "clickThroughRate": 0.19839107906664039,
      "conversionCount": 11508,
      "conversionRate": 0.13339670101658765,
      "description": "",
      "index": "atis-abtest-default",
      "noResultCount": 0,
      "searchCount": 86269,
      "trafficPercentage": 70,
      "userCount": 55501
    },
    {
      "averageClickPosition": 0,
      "clickCount": 7716,
      "clickThroughRate": 0.20869847452125934,
      "conversionCount": 5129,
      "conversionRate": 0.13872660391647734,
      "description": "",
      "index": "atis-abtest-salesRank",
      "noResultCount": 0,
      "searchCount": 36972,
      "trafficPercentage": 30,
      "userCount": 22694
    }
  ]
}

List A/B tests

Path: /2/abtests
HTTP Verb: GET
Required API Key: any key with the analytics ACL

Description:
Fetch all existing A/B tests for App that are available for the current API Key. Returns an array of metadata and metrics.

Notes:

  • This endpoint is paginated in the same way as other Analytics endpoints (i.e. using ?limit=10&offset=0).
  • As this endpoint fetches the list of all ABTests and then filters out the one that are not authorized for the given key, we do not return an error in case no ABTest is authorized for the key. We would just return an empty list of ABTests.
  • When no data has been processed, the metrics will be returned as null

Parameters:

offset
type: int
default: 0

Position of the starting record. Used for paging. 0 is the first record.

limit
type: float
default: 10

Number of records to return. +used for paging. Limit is the size of the page.

Response

clickSignificance
type: float

A/B test significance based on click data. Should be > 0.95 to be considered significant (no matter which variant is winning).

conversionSignificance
type: float

A/B test significance based on conversion data. Should be > 0.95 to be considered significant (no matter which variant is winning)

averageClickPosition
type: int

Average click position for the variant.

clickCount
type: int

Distinct click count for the variant.

clickThroughRate
type: float

Click through rate for the variant.

conversionCount
type: int

Distinct conversion count for the variant.

conversionRate
type: float

Conversion rate for the variant.

customSearchParameters
type: object

Search parameters applied to the variant.

count
type: int

Number of A/B tests found for the app.

total
type: int

Number of A/B tests retrievable.

Errors:

  • 401: AppID or API Key missing

Example:

1
2
3
4
curl -X GET \
     -H "X-Algolia-API-Key: ${API_KEY}" \
     -H "X-Algolia-Application-Id: ${APPLICATION_ID}" \
    "https://analytics.algolia.com/2/abtests?offset=${OFFSET}&limit=${LIMIT}"

Upon success, the response is 200 OK.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
{
  "abtests": [
    {
      "abTestID": 78,
      "clickSignificance": 1,
      "conversionSignificance": 0.9937,
      "createdAt": "2018-05-15T17:52:15.644906Z",
      "endAt": "2018-05-17T23:59:59Z",
      "name": "Custom Ranking salesRank Test",
      "status": "stopped",
      "variants": [
        {
          "averageClickPosition": 0,
          "clickCount": 17115,
          "clickThroughRate": 0.19839107906664039,
          "conversionCount": 11508,
          "conversionRate": 0.13339670101658765,
          "description": "",
          "index": "atis-abtest-default",
          "noResultCount": 0,
          "searchCount": 86269,
          "trafficPercentage": 70,
          "userCount": 55501
        },
        {
          "averageClickPosition": 0,
          "clickCount": 7716,
          "clickThroughRate": 0.20869847452125934,
          "conversionCount": 5129,
          "conversionRate": 0.13872660391647734,
          "description": "",
          "index": "atis-abtest-salesRank",
          "noResultCount": 0,
          "searchCount": 36972,
          "trafficPercentage": 30,
          "userCount": 22694
        }
      ]
    },
    {
      "abTestID": 80,
      "clickSignificance": 0.8187000000000001,
      "conversionSignificance": 0.6645,
      "createdAt": "2018-05-15T17:55:36.358352Z",
      "endAt": "2018-05-17T23:59:59Z",
      "name": "Searchable Attribute Manufacturer test",
      "status": "stopped",
      "variants": [
        {
          "averageClickPosition": 0,
          "clickCount": 65131,
          "clickThroughRate": 0.22,
          "conversionCount": 42808,
          "conversionRate": 0.14459719641952373,
          "description": "",
          "index": "atis-abtest-default",
          "noResultCount": 0,
          "searchCount": 296050,
          "trafficPercentage": 90,
          "userCount": 185719
        },
        {
          "averageClickPosition": 0,
          "clickCount": 7309,
          "clickThroughRate": 0.22219857724813036,
          "conversionCount": 4785,
          "conversionRate": 0.14546725846658964,
          "description": "",
          "index": "atis-abtest-manufacturer",
          "noResultCount": 0,
          "searchCount": 32894,
          "trafficPercentage": 10,
          "userCount": 20019
        }
      ]
    }
  ],
  "count": 2,
  "total": 2
}

Stop A/B test

Path: /2/abtests/{id}/stop
HTTP Verb: POST
Required API Key: any key with the setSettings ACL

Description:
Marks the A/B test as stopped. At this point, the test is over and cannot be restarted. As a result, your application is back to normal: index A will perform as usual, receiving 100% of all search requests. Associated metadata and metrics are still stored.

Parameters:

id
type: int
Required

A/B test ID.

Errors:

  • 400: Invalid signature or invalid request
  • 401: AppID or API Key missing
  • 402: Feature not enabled
  • 403: Invalid credentials
  • 404: A/B test ID not found
  • 409: A/B test was correctly deleted from the engine but the Analytics request failed (inconsistent state)
  • 422: Invalid ID format
  • 500: Internal error (possible e.g. if insertion query fails or engine query fails)

Example:

1
2
3
4
curl -X POST \
     -H "X-Algolia-API-Key: ${API_KEY}" \
     -H "X-Algolia-Application-Id: ${APPLICATION_ID}" \
    "https://analytics.algolia.com/2/abtests/#{AB_TEST_ID}/stop"

Upon success, the response is 200 OK.

Delete A/B test

Path: /2/abtests/{id}
HTTP Verb: DELETE
Required API Key: any key with the setSettings ACL

Description:
Deletes the A/B Test and removes all associated metadata & metrics.

Parameters:

id
type: int
Required

A/B test ID.

Errors:

  • 400: Invalid signature or invalid request
  • 401: AppID or API Key missing
  • 402: Feature not enabled
  • 403: Invalid credentials
  • 404: A/B test ID not found
  • 409: A/B test was correctly deleted from the engine but the Analytics request failed (inconsistent state)
  • 422: Invalid payload, A/B test already running on one of the indices
  • 500: Internal error (possible e.g. if insertion query fails or engine query fails)

Example:

1
2
3
4
curl -X DELETE \
     -H "X-Algolia-API-Key: ${API_KEY}" \
     -H "X-Algolia-Application-Id: ${APPLICATION_ID}" \
    "https://analytics.algolia.com/2/abtests/${AB_TEST_ID}"

Upon success, the response is 200 OK.

Did you find this page helpful?