Tools / Crawler / requestOptions

requestOptions

Type: object
Parameter syntax
requestOptions: {
   proxy: 'http://proxy'
   timeout: 10000
   retries: 2
   headers: {
     ...
   }
}

About this parameter

Modify all crawler’s requests behavior.

The Crawler only accepts headers from our list of allowed headers.

The Cookie header will be replaced by the cookie fetched in login.

Examples

1
2
3
4
5
6
7
8
9
10
11
{
  requestOptions: {
    proxy: 'http://algolia:password@1.1.1.1:8080',
    timeout: 10000,
    retries: 2,
    headers: {
      Authorization: 'Bearer Aerehdf==',
      Cookie: 'session=1234;',
    }
  }
}

Parameters

proxy
type: string (URL)
Optional

Set a proxy for your crawler’s requests.

timeout
type: number
default: 30000
Optional

Set a timeout (in milliseconds) to your crawler’s requests.

retries
type: number
default: 3
Optional

Set a maximum retry to your crawler’s requests.

headers
type: Object
Optional

Specify headers to be sent in the crawler’s requests.

Supported headers:

  • Accept-Language
  • Authorization
  • Cookie

Did you find this page helpful?