Tools
/
Crawler
/
APIs
/
Configuration
Jan 17, 2022
Crawler: IgnoreRobotsTxtRules
Type:
boolean
Default:
false
Parameter syntax
ignoreRobotsTxtRules: true|false
About this parameter
When set to true
, the crawler will ignore rules set in your robots.txt.
Examples
Copy
1
2
3
{
ignoreRobotsTxtRules: true,
}
Did you find this page helpful?