Crawler API overview

You can manage and run your crawlers from the dashboard, using the Algolia CLI, or with the Crawler API.

The Crawler API lets you:

• Change the state of crawlers, such as pausing crawl schedules or testing the crawler with specific URLs.
• Manage your crawlers: create, rename, and delete them.
• List your registered domains.
• Examine the status of crawler tasks.
• Create and update your crawler configuration to specify which URLs to crawl, when to crawl, how to extract records from the crawl, and where to index the extracted records.