Tools / Crawler / APIs / Configuration

Crawler: RenderJavaScript

Type: boolean | string[] | object
Default: false
Parameter syntax
renderJavaScript: true|false|[
  'http://www.example.com',
  ...
] | {
  enabled: boolean,
  waitTime?: {
    min?: number,
    max?: number
  },
  adblock?: boolean,
  patterns: [
    'http://www.algolia.com/blog/**'
  ]
}

About this parameter

When true, all web pages are rendered with a chrome headless browser. The crawler will use the rendered HTML.

Because rendering JavaScript-based web pages is slower than crawling regular HTML pages, you can apply this setting to a specified list of micromatch URL patterns. These patterns can include negations and wildcards.

With this setting enabled, JavaScript is executed on the web page by a headless Chrome browser. Because a lot of websites have infinite refreshes and updates, the browser has a timeout of 20 seconds by default. Change this value with renderJavascript.waitTime.

Enabling this feature can lead to inconsistent records across crawls, depending on the browser load and the website speed.

Make sure your JavaScript-enabled pages load fast enough for the crawler.

Examples

1
2
3
{
  renderJavaScript: true,
}
1
2
3
4
5
{
  renderJavaScript: [
    'http://www.mysite.com/dynamic-pages/**',
  ],
}
1
2
3
4
5
6
7
8
9
10
11
12
13
{
  renderJavaScript: {
    enabled: true,
    waitTime: {
      min: 7000,
      max: 15000,
    }
    adblock: true,
    patterns: [
      'http://www.mysite.com/dynamic-pages/**',
    ],
  }
}

Parameters

renderJavascript

enabled
type: boolean
Optional

Determines the activation of the JavaScript rendering feature.

patterns
type: string[]
Optional

Based on URL patterns, determines which web pages to render with JavaScript. If unspecified, and enabled is set to true, all web pages are rendered with JavaScript.

waitTime
type: object
Optional

Determines the shortest and longest wait time before considering a page as rendered.

adblock
type: boolean
default: false
Optional

Whether to activate the custom ad blocker or not. It blocks most ads and tracking scripts, but can break some websites.

renderJavascript ➔ waitTime

min
type: number
default: 0
Optional

If the page runs faster than this minimum execution time, the browser remains open at least this long.

max
type: number
default: 20000
Optional

At this maximum execution time threshold, the execution stops and the HTML is processed as is.

Did you find this page helpful?