Skip to main content
  • Type: number
  • Required
A higher number means more crawls per second. This setting uses the following formula: 
MAX ( urls_added_in_the_last_second, urls_currently_being_processed ) <= rateLimit
If retrieving, processing, and uploading URLs is taking less than a second, your crawler processes rateLimit URLs per second. However, if each URL takes 4 seconds on average to process, your crawler processes rateLimit / 4 pages per second. Start with a low value and update it if you need faster crawls. High rateLimit values can have a large impact on your bandwidth usage and server resource consumption.

​
Examples

JavaScript
{
  rateLimit: 5,
}