Tools
/
Crawler
/
APIs
/
Configuration
Jan 17, 2022
Crawler: RequestOptions
Type:
object
Parameter syntax
requestOptions: {
proxy: 'http://proxy'
timeout: 10000
retries: 2
headers: {
...
}
}
About this parameter
Modify all crawler’s requests behavior.
The Crawler only accepts headers from the list of allowed headers.
The Cookie header is replaced by the cookie fetched in login.
Examples
Copy
1
2
3
4
5
6
7
8
9
10
11
{
requestOptions: {
proxy: 'http://algolia:password@1.1.1.1:8080',
timeout: 10000,
retries: 2,
headers: {
Authorization: 'Bearer Aerehdf==',
Cookie: 'session=1234;',
}
}
}
Parameters
proxy
|
type: string (URL)
Optional
Set a proxy for your crawler’s requests. |
timeout
|
type: number
default: 30000
Optional
Set a timeout (in milliseconds) to your crawler’s requests. |
retries
|
type: number
default: 2
Optional
Set a maximum retry to your crawler’s requests. |
headers
|
type: Object
Optional
Specify headers to be sent in the crawler’s requests. Supported headers:
|
Did you find this page helpful?
