API Reference / Crawler Configuration API / login
Type: object
Parameter syntax
login: {
  fetchRequest: {
    url: 'your_url',
    requestOptions: {
      ...
    }
  }
}

About this parameter

This property defines how the crawler acquires a session cookie.

The crawler extracts the Set-Cookie response header from the login page and sends that cookie when crawling all pages of the website defined in the configuration.

This cookie is only fetched at the beginning of each complete crawl. If it expires, we won’t renew it automatically.

Examples

1
2
3
4
5
6
7
8
9
10
11
12
13
{
  login: {
    fetchRequest: {
      url: `http://example.com/secure/login-with-post`,
      requestOptions: {
        method: 'POST',
        headers: { 'Content-Type': 'application/x-www-form-urlencoded' },
        body: 'id=my-id&password=my-password',
        timeout: 5000 // in milliseconds
      }
    }
  }
}

Parameters

fetchRequest

url
type: string
Required

The URL to target.

requestOptions
type: Object

This object is passed to our extended version of the request library.

fetchRequest ➔ requestOptions

method
type: string
default: GET

The HTTP method to use.

headers
type: object
default: {}

HTTP headers to pass.

body
type: string

The body of the request.

timeout
type: number

Time to wait before aborting the request (in milliseconds).

requestOptions ➔ headers

Content-Type
type: string
Authorization
type: string
type: string

Did you find this page helpful?