API Reference / REST API / Crawler REST API
Sep. 26, 2019

Crawler REST API

Overview

The Crawler REST API is only available if you have access to Crawler.

The Crawler REST API lets your interact directly with your crawlers.

All API calls go through the https://crawler.algolia.com/api/1 domain.

Request Format

Authentication is done via basic authentication.

Response Format

The response format for all requests is a JSON object.

The success or failure of an API call is indicated by its HTTP status code. A 2xx status code indicates success, whereas a 4xx status code indicates failure. When a request fails, the response body is still JSON, but contains a message field which you can use for debugging.

Crawler Endpoints

Quick Reference

Verb Path Method

POST

/crawlers

Create a crawler

GET

/crawlers

Get available crawlers

GET

/crawlers/{id}

Get a crawler

PATCH

/crawlers/{id}

Partially update a crawler

PATCH

/crawlers/{id}/config

Partially update a crawler's configuration

POST

/crawlers/{id}/run

Run a crawler

POST

/crawlers/{id}/pause

Pause a crawler's run

POST

/crawlers/{id}/reindex

Reindex with a crawler

GET

/crawlers/{id}/stats/urls

Get statistics on a crawler

POST

/crawlers/{id}/test

Test a URL on a crawler

GET

/crawlers/{id}/tasks/{tid}

Get status of a task

Create a crawler

Path: /crawlers
HTTP Verb: POST

Description:
Create a new crawler with the given configuration.

Errors:

  • 400: Bad request or request argument.
  • 401: Authorization information is missing or invalid.
  • 403: The user doesn’t have access rights to the specified crawler, or the crawler doesn’t exist.

Example:

1
2
3
4
curl -X POST --user $CRAWLER_USER_ID:$CRAWLER_API_KEY "https://crawler.algolia.com/api/1/crawlers/" \ 
  -H "accept: application/json" \
  -H "Content-Type: application/json" \
  -d @create_crawler.json

This is the create_crawler.json file used in the curl call.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
{
  "name": "Algolia Website",
  "config": {
    "appId": "YOUR_APP_ID",
    "apiKey": "YOUR_API_KEY",
    "indexPrefix": "crawler_",
    "rateLimit": 8,
    "startUrls": [
      "https://www.algolia.com"
    ],
    "actions": [
      {
        "indexName": "algolia_website",
        "pathsToMatch": [
          "https://www.algolia.com/**"
        ],
        "selectorsToMatch": [
          ".products",
          "!.featured"
        ],
        "fileTypesToMatch": [
          "html",
          "pdf"
        ],
        "recordExtractor": {
          "__type": "function",
          "source": "() => {}"
        }
      }
    ]
  }
}

When the request is successful, the HTTP response is a 200 OK and returns the ID of the created crawler:

1
2
3
{
  "id": "x000x000-x00x-00xx-x000-000000000x00"
}

Get available crawlers

Path: /crawlers
HTTP Verb: GET

Description:
Get a list of your available crawlers and pagination information

Errors:

  • 400: Bad request or request argument.
  • 401: Authorization information is missing or invalid.
  • 403: The user doesn’t have access rights to the specified crawler, or the crawler doesn’t exist.

Example:

1
curl -X GET -G --user $CRAWLER_USER_ID:$CRAWLER_API_KEY "https://crawler.algolia.com/api/1/crawlers"

When the request is successful, the HTTP response is 200 OK and returns a list of your available crawlers:

1
2
3
4
5
6
7
8
9
10
11
{
  "items": [
    {
      "id": "e0f6db8a-24f5-4092-83a4-1b2c6cb6d809",
      "name": "My Crawler"
    }
  ],
  "itemsPerPage": 20,
  "page": 1,
  "total": 100
}

Get a crawler

Path: /crawlers/{id}
HTTP Verb: GET

Description:
Get information about the specified crawler and its settings.

Parameters:

id
type: string
Required

The ID of your targeted crawler.

Errors:

  • 400: Bad request or request argument.
  • 401: Authorization information is missing or invalid.
  • 403: The user doesn’t have access rights to the specified crawler, or the crawler doesn’t exist.

Example:

1
curl -X GET -G --user $CRAWLER_USER_ID:$CRAWLER_API_KEY "https://crawler.algolia.com/api/1/crawlers/$CRAWLER_ID?withConfig=true"

When the request is successful, the HTTP response is 200 OK and returns information on the specified crawler:

1
2
3
4
5
6
7
8
9
10
11
12
13
{
   "name": "Algolia Blog",
   "createdAt": "2019-08-22T08:17:09.680Z",
   "updatedAt": "2019-09-02T09:57:33.968Z",
   "running": false,
   "reindexing": false,
   "blocked": false,
   "lastReindexStartedAt": "2019-08-29T07:56:51.682Z",
   "lastReindexEndedAt": "2019-08-29T07:57:35.991Z"
   "config": {
     // YOUR CRAWLER'S CONFIGURATION
   }
}

Partially update a crawler

Path: /crawlers/{id}
HTTP Verb: PATCH

Description:
Update a crawler’s configuration or change its name.

Parameters:

id
type: string
Required

The ID of your targeted crawler.

Errors:

  • 400: Bad request or request argument.
  • 401: Authorization information is missing or invalid.
  • 403: The user doesn’t have access rights to the specified crawler, or the crawler doesn’t exist.

Example:

1
2
curl -X PATCH --user $CRAWLER_USER_ID:$CRAWLER_API_KEY "https://crawler.algolia.com/api/1/crawlers/$CRAWLER_ID/" \
  -d @crawler_patch.json

The crawler_patch.json file:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
{
  "name": "New config name",
  "config": {
    "appId": "YOUR_APP_ID",
    "apiKey": "YOUR_API_KEY",
    "indexPrefix": "crawler_",
    "rateLimit": 8,
    "startUrls": [
      "https://www.algolia.com"
    ],
    "actions": [
      {
        "indexName": "algolia_website",
        "pathsToMatch": [
          "https://www.algolia.com/**"
        ],
        "selectorsToMatch": [
          ".products",
          "!.featured"
        ],
        "fileTypesToMatch": [
          "html",
          "pdf"
        ],
        "recordExtractor": {
          "__type": "function",
          "source": "() => {}"
        }
      }
    ]
  }
}

When the request is successful, the HTTP response is a 200 OK and returns the taskId of your request:

1
2
3
{
  "taskId": "e0f6db8a-24f5-4092-83a4-1b2c6cb6d809"
}

Partially update a crawler’s configuration

Path: /crawlers/{id}/config
HTTP Verb: PATCH

Description:
Update parts of a crawler’s configuration.

Parameters:

id
type: string
Required

The ID of your targeted crawler.

Errors:

  • 400: Bad request or request argument.
  • 401: Authorization information is missing or invalid.
  • 403: The user doesn’t have access rights to the specified crawler, or the crawler doesn’t exist.

Example:

1
2
curl -X PATCH --user $CRAWLER_USER_ID:$CRAWLER_API_KEY "https://crawler.algolia.com/api/1/crawlers/$CRAWLER_ID/" \                              [ NORMAL ]
  -d @crawler_config_patch.json

The crawler_config_patch.json file:

1
2
3
4
5
6
7
{
  "rateLimit": 8,
  "startUrls": [
    "https://www.algolia.com",
    "https://www.algolia.com/blog"
  ],
}

When the request is successful, the HTTP response is a 200 OK and returns the taskId of your request:

1
2
3
{
  "taskId": "e0f6db8a-24f5-4092-83a4-1b2c6cb6d809"
}

Run a crawler

Path: /crawlers/{id}/run
HTTP Verb: POST

Description:
Request the specified crawler to run.

Parameters:

id
type: string
Required

The ID of your targeted crawler.

Errors:

  • 401: Authorization information is missing or invalid.
  • 403: The user doesn’t have access rights to the specified crawler, or the crawler doesn’t exist.

Example:

1
curl -X POST --user $CRAWLER_USER_ID:$CRAWLER_API_KEY "https://crawler.algolia.com/api/1/crawlers/$CRAWLER_ID/run"

When the request is successful, the HTTP response is a 200 OK and returns the task ID of the crawl:

1
2
3
{
  "taskId": "e0f6db8a-24f5-4092-83a4-1b2c6cb6d809"
}

Pause a crawler’s run

Path: /crawlers/{id}/pause
HTTP Verb: POST

Description:
Request the specified crawler to pause its eecution.

Parameters:

id
type: string
Required

The ID of your targeted crawler.

Errors:

  • 400: Bad request or request argument.
  • 401: Authorization information is missing or invalid.
  • 403: The user doesn’t have access rights to the specified crawler, or the crawler doesn’t exist.

Example:

1
curl -X POST --user $CRAWLER_USER_ID:$CRAWLER_API_KEY "https://crawler.algolia.com/api/1/crawlers/$CRAWLER_ID/pause"

When the request is successful, the HTTP response is a 200 OK and returns the task ID of your request:

1
2
3
{
  "taskId": "e0f6db8a-24f5-4092-83a4-1b2c6cb6d809"
}

Reindex with a crawler

Path: /crawlers/{id}/reindex
HTTP Verb: POST

Description:
Request the specified crawler to start restart crawling.

Parameters:

id
type: string
Required

The ID of your targeted crawler.

Errors:

  • 401: Authorization information is missing or invalid.
  • 403: The user doesn’t have access rights to the specified crawler, or the crawler doesn’t exist.

Example:

1
curl -X POST --user $CRAWLER_USER_ID:$CRAWLER_API_KEY "https://crawler.algolia.com/api/1/crawlers/$CRAWLER_ID/reindex"

When the request is successful, the HTTP response is a 200 OK and returns the task ID of the crawl:

1
2
3
{
  "taskId": "e0f6db8a-24f5-4092-83a4-1b2c6cb6d809"
}

Get statistics on a crawler

Path: /crawlers/{id}/stats/urls
HTTP Verb: GET

Description:
Get a summary of the current status of crawled URLs for the specified crawler.

Parameters:

id
type: string
Required

The ID of your targeted crawler.

Errors:

  • 401: Authorization information is missing or invalid.
  • 403: The user doesn’t have access rights to the specified crawler, or the crawler doesn’t exist.

Example:

1
curl -X GET -G --user $CRAWLER_USER_ID:$CRAWLER_API_KEY "https://crawler.algolia.com/api/1/crawlers/$CRAWLER_ID/stats/urls"

When the request is successful, the HTTP response is a 200 OK and returns URL statistics for your specified crawler’s last crawl:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
{
  "count":129,
  "data":[
    {
      "reason":"success",
      "status":"DONE",
      "category":"success",
      "readable":"Success",
      "count":127
    },
    {
      "reason":"http_not_found",
      "status":"SKIPPED",
      "category":"fetch",
      "readable":"HTTP Not Found (404)",
      "count":1
    },
    {
      "reason":"network_error",
      "status":"FAILED",
      "category":"fetch",
      "readable":"Network error",
      "count":1
    }
  ]
}

Test a URL on a crawler

Path: /crawlers/{id}/test
HTTP Verb: POST

Description:
Test an URL against the given crawler’s configuration and see what will be processed. You can also override parts of the configuration to try your changes before updating the configuration.

Parameters:

id
type: string
Required

The ID of your targeted crawler

Errors:

  • 400: Bad request or request argument.
  • 401: Authorization information is missing or invalid.
  • 403: The user doesn’t have access rights to the specified crawler, or the crawler doesn’t exist.

Example:

1
2
3
4
curl -X POST --user $CRAWLER_USER_ID:$CRAWLER_API_KEY "https://crawler.algolia.com/api/1/crawlers/$CRAWLER_ID/test" \                                           [ INSERT
  -H "accept: application/json" \
  -H "Content-Type: application/json" \
  -d @test_crawler.json

This is the test_crawler.json file used in curl call.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
{
  "url": "https://www.algolia.com/blog",
  "config": {
    "appId": "YOUR_APP_ID",
    "apiKey": "YOUR_API_KEY",
    "indexPrefix": "crawler_",
    "rateLimit": 8,
    "startUrls": [
      "https://www.algolia.com"
    ],
    "actions": [
      {
        "indexName": "algolia_website",
        "pathsToMatch": [
          "https://www.algolia.com/**"
        ],
        "selectorsToMatch": [
          ".products",
          "!.featured"
        ],
        "fileTypesToMatch": [
          "html",
          "pdf"
        ],
        "recordExtractor": {
          "__type": "function",
          "source": "() => {}"
        }
      }
    ]
  }
}

When the request is successful, the HTTP response is a 200 OK and returns the ID of the created crawler:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
{
  "startDate": "2019-05-21T09:04:33.742Z",
  "endDate": "2019-05-21T09:04:33.923Z",
  "logs": [
    [
      "Processing url 'https://www.algolia.com/blog'"
    ]
  ],
  "records": [
    {
      "indexName": "testIndex",
      "records": [
        {
          "objectID": "https://www.algolia.com/blog",
          "numberOfLinks": 2
        }
      ],
      "recordsPerExtractor": [
        {
          "index": 0,
          "type": "custom",
          "records": [
            {
              "objectID": "https://www.algolia.com/blog"
            }
          ]
        }
      ]
    }
  ],
  "links": [
    "https://blog.algolia.com/challenging-migration-heroku-google-kubernetes-engine/",
    "https://blog.algolia.com/tale-two-engines-algolia-unity/"
  ],
  "externalData": {
    "dataSourceId1": {
      "data1": "val1",
      "data2": "val2"
    },
    "dataSourceId2": {
      "data1": "val1",
      "data2": "val2"
    }
  },
  "error": {}
}

Get status of a task

Path: /crawlers/{id}/tasks/{tid}
HTTP Verb: GET

Description:
Get the status of a specific task

Parameters:

id
type: string
Required

The ID of the targeted crawler.

tid
type: string
Required

The ID of the targeted task.

Errors:

  • 400: Bad request or request argument.
  • 401: Authorization information is missing or invalid.
  • 403: The user doesn’t have access rights to the specified crawler, or the crawler doesn’t exist.

Example:

1
  curl -X GET -G --user $CRAWLER_USER_ID:$CRAWLER_API_KEY "https://crawler.algolia.com/api/1/crawlers/$CRAWLER_ID/tasks/18bf6357-fbad-42b2-9a0f-d685e25a24f9"

When the request is successful, the HTTP response is 200 OK and returns whether the task is pending or not:

1
2
3
{
  "pending": false
}

Did you find this page helpful?