Skip to main content

Documentation Index

Fetch the complete documentation index at: https://algolia.com/llms.txt

Use this file to discover all available pages before exploring further.

Usage
algolia crawler crawl <crawler_id> --urls <url>... [flags]

​
Examples

Crawl the URLs “https://www.example.com” and “https://www.example2.com/” for the crawler with the ID “my-crawler”: 
algolia crawler crawl my-crawler --urls https://www.example.com,https://www.example2.com/
Crawl the URLs “https://www.example.com” and “https://www.example2.com/” for the crawler with the ID “my-crawler” and save them in the configuration: 
algolia crawler crawl my-crawler --urls https://www.example.com,https://www.example2.com/ --save
Crawl the URLs “https://www.example.com” and “https://www.example2.com/” for the crawler with the ID “my-crawler” and don’t save them in the configuration: 
algolia crawler crawl my-crawler --urls https://www.example.com,https://www.example2.com/ --save=false

​
Flags

​
-s, --save
When true, the URLs are added to your %[1]sextraUrls%[1]s (unless present in %[1]sstartUrls%[1]s or %[1]ssitemaps%[1]s). When false, the URLs aren’t added. When unspecified, the URLs are added to your %[1]sextraUrls%[1]s (unless present in %[1]sstartUrls%[1]s or %[1]ssitemaps%[1]s or they weren’t indexed during the preceding reindex).
​
-u, --urls
The URLs to crawl (maximum 50).
Last modified on May 12, 2026