Tools / Crawler / Getting started

Configure a crawler with the visual UI

To change the most commonly used crawler configuration settings, use the visual UI. To access the visual UI, click Configuration in the Crawler’s left-hand menu.

For advanced configuration options, use the editor.

Changes made in the visual UI are instantly replicated in the editor and vice versa. You can track all changes in the editor’s Configuration History.

Visual UI for crawler configuration

Configuration settings

Use the visual UI to change the following settings.

Setting Description Editor equivalent
Start URL Specifies which URLs the crawler should start with startUrls
Disable limit/Max URLs Limit the number of URLs the crawler can process maxUrls
Crawler Schedule Sets automatic run times for the crawler schedule
URL Exclusion Rules Excludes specific sections or types of content from the crawl exclusionPatterns
Record Loss Policy Stops the crawler if it finds significantly fewer records than the previous crawl maxLostRecordsPercentage
Max Failed URLs Stops the crawler if a specified number of pages fail to crawl maxFailedUrls
Website Authentication Adds authentication details for password protected sites login
Index Backup Saves a copy of previously crawled content saveBackup
Did you find this page helpful?