Crawler: IgnoreNoIndex

About this parameter

Whether the Crawler should extract records from a page whose robots meta tag contains noindex or none.

When true, the Crawler ignores the noindex directive of the robots meta tag.

Examples

1
2
3
{
  ignoreNoIndex: true,
}