API Reference / Crawler Configuration API / requestOptions

requestOptions

Type: object
Parameter syntax
requestOptions: {
   proxy: 'http://proxy'
   headers: {
     ...
   }
}

About this parameter

Set a proxy and headers for the crawler’s web requests.

The Crawler only accepts headers from our list of allowed headers.

The Cookie header will be replaced by the cookie fetched in login.

Examples

1
2
3
4
5
6
7
8
9
{
  requestOptions: {
    proxy: 'http://algolia:password@1.1.1.1:8080',
    headers: {
      Authorization: 'Bearer Aerehdf==',
      Cookie: 'session=1234;',
    }
  }
}

Parameters

proxy
type: string (URL)
Optional

Set a proxy for your crawler’s requests.

headers
type: Object
Optional

Specify headers to be sent in the crawler’s requests.

Supported headers:

  • Accept-Language
  • Authorization
  • Cookie

Did you find this page helpful?