Tools
/
Crawler
/
Getting started
Nov 18, 2024
Configure a crawler with the visual UI
To change the most commonly used crawler configuration settings, use the visual UI. To access the visual UI, click Configuration in the Crawler’s left-hand menu.
For advanced configuration options, use the editor.
Changes made in the visual UI are instantly replicated in the editor and vice versa. You can track all changes in the editor’s Configuration History.
Configuration settings
Use the visual UI to change the following settings.
Setting | Description | Editor equivalent |
---|---|---|
Start URL | Specifies which URLs the crawler should start with | startUrls |
Disable limit/Max URLs | Limit the number of URLs the crawler can process | maxUrls |
Crawler Schedule | Sets automatic run times for the crawler | schedule |
URL Exclusion Rules | Excludes specific sections or types of content from the crawl | exclusionPatterns |
Record Loss Policy | Stops the crawler if it finds significantly fewer records than the previous crawl | maxLostRecordsPercentage |
Max Failed URLs | Stops the crawler if a specified number of pages fail to crawl | maxFailedUrls |
Website Authentication | Adds authentication details for password protected sites | login |
Index Backup | Saves a copy of previously crawled content | saveBackup |
Did you find this page helpful?