Tools
/
Crawler
/
login
Jan. 21, 2021
login
Type:
object
Parameter syntax
login: {
fetchRequest: {
url: 'your_url',
requestOptions: {
...
}
}
}
About this parameter
This property defines how the crawler acquires a session cookie.
The crawler extracts the Set-Cookie response header from the login page and sends that cookie when crawling all pages of the website defined in the configuration.
This cookie is only fetched at the beginning of each complete crawl. If it expires, we won’t renew it automatically.
Examples
Copy
1
2
3
4
5
6
7
8
9
10
11
12
13
{
login: {
fetchRequest: {
url: `http://example.com/secure/login-with-post`,
requestOptions: {
method: 'POST',
headers: { 'Content-Type': 'application/x-www-form-urlencoded' },
body: 'id=my-id&password=my-password',
timeout: 5000 // in milliseconds
}
}
}
}
Parameters
fetchRequest
url
|
type: string
Required
The URL to target. |
requestOptions
|
type: Object
This object is passed to our extended version of the request library. |
fetchRequest ➔ requestOptions
method
|
type: string
default: GET
The HTTP method to use. |
headers
|
type: object
default: {}
HTTP headers to pass. |
body
|
type: string
The body of the request. |
timeout
|
type: number
Time to wait before aborting the request (in milliseconds). |
requestOptions ➔ headers
Content-Type
|
type: string
|
Authorization
|
type: string
|
Cookie
|
type: string
|
