Tools
/
Crawler
/
APIs
/
Configuration
Jan 17, 2022
Crawler: MaxUrls
Type:
number
Default:
1,000,000
Parameter syntax
maxUrls: max_urls
About this parameter
Limits the number of URLs your crawler can process.
Useful for demoing or preventing infinite link holes in the website structure.
maxUrls does not guarantee consistent indexing between crawls. The crawler processes in parallel, so discovered URLs can be processed in different orders for different crawls.
Examples
Copy
1
2
3
{
maxUrls: 30,
}
