Configure crawlers with the visual UI - Algolia

To access the visual UI, click Configuration in the Crawler’s left-hand menu. For advanced configuration options, use the editor. The visual UI and the editor stay in sync. A change in one shows up right away in the other. You can track all changes in the editor’s Configuration History.

Screenshot of the 'Crawler Configuration' page with 'Crawler Schedule,' 'URL Exclusion Rules,' and 'Record Loss Policy' sections, each with an 'Edit' button.

Configuration settings

Use the visual UI to change the following settings.

Setting	Description	Editor equivalent
Start URL	Specifies which URLs the crawler should start with. See also: Where to start crawling	`startUrls`
Disable limit/Max URLs	Limit the number of URLs the crawler can process. See also: Maximum URL limit	`maxUrls`
Crawler Schedule	Sets automatic run times for the crawler	`schedule`
URL Exclusion Rules	Excludes specific sections or types of content from the crawl. See also: Exclude and include content	`exclusionPatterns`
Record Loss Policy	Stops the crawler if it finds significantly fewer records than the previous crawl. See also: Safety checks	`maxLostRecordsPercentage`
Max Failed URLs	Stops the crawler if a specified number of pages fail to crawl. See also: Safety checks	`maxFailedUrls`
Website Authentication	Adds authentication details for password protected sites	`login`
Index Backup	Saves a copy of previously crawled content	`saveBackup`

Last modified on September 16, 2026