API Reference / Crawler Configuration API / maxUrls
Type: number
Default: 15,000,000
Parameter syntax
maxUrls: max_urls

About this parameter

Limits the number of URLs your crawler can process.

Useful for demoing or preventing infinite link holes in the website structure.

maxUrls does not guarantee consistent indexing between crawls. The crawler processes in parallel, so discovered URLs can be processed in different orders for different crawls.

Examples

1
2
3
{
  maxUrls: 30,
}

Did you find this page helpful?