Tools / Crawler / ignoreNoIndex

ignoreNoIndex

Type: boolean
Default: false
Parameter syntax
ignoreNoIndex: true|false

About this parameter

Whether the Crawler should extract records from a page whose robots meta tag contains noindex or none.

When true, the Crawler ignores the noindex directive of the robots meta tag.

Examples

1
2
3
{
  ignoreNoIndex: true,
}

Did you find this page helpful?