FAQ / Basics / What are the Crawler technical limitations?

What are the Crawler technical limitations?

In addition to any plan limits, the Crawler is subject to the following technical limits:

Data volume limitations for crawler free trial only

Number of URLs crawled (per month) 100K URLs

Data volume limitations

Size per document 50 MB
Crawling refresh/recrawl per day Manual: 200
Automatic: once

Crawler Console Limitations

Number of statistics retrieved from analytics tool Only top 100K pages
Number of CSV files/lines imported as an external sources
(per crawling operation)
5 million times

Data Retrieval Frequency

The minimum period of time between data updates (crawls) is 24 hours. Real-time indexing is not guaranteed.

Application Restrictions

The crawler needs to access your website in order to index data to Algolia. You will need to ensure it is granted the appropriate access rights (whitelisting, authorizations).

Metadata Limitations

As the crawler is limited by the data it can access, you may need to inject additional metrics besides what’s currently available on your website in order to tailor the search experience to your business needs.

Google Analytics Limitations

In compliance with Google Analytics restrictions, the crawler is limited to 10,000 Google Analytics API requests per day.

Did you find this page helpful?