Algolia DevCon
Oct. 2–3 2024, virtual.
Tools / Crawler / APIs / Configuration

Crawler: IgnoreRobotsTxtRules

Type: boolean
Default: false
Parameter syntax
ignoreRobotsTxtRules: true|false

About this parameter

When set to true, the crawler will ignore rules set in your robots.txt.

Examples

1
2
3
{
  ignoreRobotsTxtRules: true,
}
Did you find this page helpful?