What are the Crawler technical limitations?
In addition to any plan limits, the Crawler is subject to the following technical limits:
Data volume limitations for crawler free trial only
|Number of URLs crawled (per month)||100K URLs|
Data volume limitations
|Size per document||10 MB|
|Crawling refresh/recrawl per day||Manual: 100
Crawler Console Limitations
|Number of statistics retrieved from analytics tool||Only top 100K pages|
|Number of CSV files/lines imported as an external sources
(per crawling operation)
Data Retrieval Frequency
The minimum period of time between data updates (crawls) is 24 hours. Real-time indexing is not guaranteed.
The crawler needs to access your website in order to index data to Algolia. You will need to ensure it is granted the appropriate access rights (whitelisting, authorizations).
As the crawler is limited by the data it can access, you may need to inject additional metrics besides what’s currently available on your website in order to tailor the search experience to your business needs.
Google Analytics Limitations
In compliance with Google Analytics restrictions, the crawler is limited to 10,000 Google Analytics API requests per day.