> ## Documentation Index
> Fetch the complete documentation index at: https://algolia.com/llms.txt
> Use this file to discover all available pages before exploring further.

# rateLimit

> Number of concurrent tasks per second for this crawler.

* **Type**: `number`
* **Required**

A higher number means more crawls per second.
This setting uses the following formula:

```txt theme={"system"}
MAX ( urls_added_in_the_last_second, urls_currently_being_processed ) <= rateLimit
```

If retrieving, processing, and uploading URLs is taking less than a second,
your crawler processes `rateLimit` URLs per second.
However, if each URL takes 4 seconds on average to process,
your crawler processes `rateLimit / 4` pages per second.

Start with a low value and update it if you need faster crawls.
High `rateLimit` values can have a large impact on your bandwidth usage and server resource consumption.

## Examples

```js JavaScript icon=code theme={"system"}
{
  rateLimit: 5,
}
```