API Reference / Crawler Configuration API / ignoreRobotsTxtRules

ignoreRobotsTxtRules

Type: boolean
Default: false
Parameter syntax
ignoreRobotsTxtRules: true|false

About this parameter

When set to true, the crawler will ignore rules set in your robots.txt.

Examples

1
2
3
{
  ignoreRobotsTxtRules: true,
}

Did you find this page helpful?