Configure crawlers with the visual UI

To access the visual UI, click Configuration in the Crawler’s left-hand menu. For advanced configuration options, use the editor. Changes made in the visual UI are instantly replicated in the editor and vice versa. You can track all changes in the editor’s Configuration History.

Screenshot of the 'Crawler Configuration' page with 'Crawler Schedule,' 'URL Exclusion Rules,' and 'Record Loss Policy' sections, each with an 'Edit' button.

Configuration settings

Use the visual UI to change the following settings.

Setting	Description	Editor equivalent
Start URL	Specifies which URLs the crawler should start with. See also: Where to start crawling	`startUrls`
Disable limit/Max URLs	Limit the number of URLs the crawler can process. See also: Maximum URL limit	`maxUrls`
Crawler Schedule	Sets automatic run times for the crawler	`schedule`
URL Exclusion Rules	Excludes specific sections or types of content from the crawl. See also: Exclude and include content	`exclusionPatterns`
Record Loss Policy	Stops the crawler if it finds significantly fewer records than the previous crawl. See also: Safety checks	`maxLostRecordsPercentage`
Max Failed URLs	Stops the crawler if a specified number of pages fail to crawl. See also: Safety checks	`maxFailedUrls`
Website Authentication	Adds authentication details for password protected sites	`login`
Index Backup	Saves a copy of previously crawled content	`saveBackup`

Last modified on May 19, 2026

Create new crawlers

Configure crawlers with the editor

Get started

Extract data

Enrich data

Troubleshooting

APIs

Netlify plugin

Configure crawlers with the visual UI

Configuration settings

​Configuration settings

Configuration settings