algolia crawler
Manage your Algolia crawlers.
algolia crawler crawl
algolia crawler crawl <crawler_id> --urls <url>... [flags]
Crawl specific URLs.
Examples
1
2
3
4
5
6
7
8
9
# Crawl the URLs "https://www.example.com" and "https://www.example2.com/" for the crawler with the ID "my-crawler"
$ algolia crawler crawl my-crawler --urls https://www.example.com,https://www.example2.com/
# Crawl the URLs "https://www.example.com" and "https://www.example2.com/" for the crawler with the ID "my-crawler" and save them in the configuration
$ algolia crawler crawl my-crawler --urls https://www.example.com,https://www.example2.com/ --save
# Crawl the URLs "https://www.example.com" and "https://www.example2.com/" for the crawler with the ID "my-crawler" and don't save them in the configuration
$ algolia crawler crawl my-crawler --urls https://www.example.com,https://www.example2.com/ --save=false
Flags
-
-s,--save -
When true, the URLs are added to your extraUrls (unless present in startUrls or sitemaps).
When false, the URLs aren’t added.
When unspecified, the URLs are added to your extraUrls (unless present in startUrls or sitemaps or they weren’t indexed during the preceding reindex). -
-u,--urls -
The URLs to crawl (maximum 50).
algolia crawler create
algolia crawler create <name> -F <file> [flags]
Create a crawler.
Examples
1
2
3
4
5
6
# Create a crawler named "my-crawler" with the configuration in the file "config.json"
$ algolia crawler create my-crawler -F config.json
# Create a crawler from another crawler's configuration
$ algolia crawler get another-crawler --config-only | algolia crawler create my-crawler -F -
Flags
-
-F,--file -
Path to the configuration file (use “-“ to read from standard input)
algolia crawler get
algolia crawler get <crawler_id> [flags]
Get a crawler.
Examples
1
2
3
4
5
6
# Get the crawler with the ID "my-crawler"
$ algolia crawler get my-crawler
# Get the crawler with the ID "my-crawler" and display only its configuration
$ algolia crawler get my-crawler --config-only
Flags
-
-c,--config-only -
Display only the crawler configuration
algolia crawler list
algolia crawler list [flags]
List crawlers.
Examples
1
2
3
4
5
6
7
8
9
# List all crawlers
$ algolia crawler list
# List crawlers with the name "my-crawler"
$ algolia crawler list --name my-crawler
# List crawlers with the appID "my-app-id"
$ algolia crawler list --app-id my-app-id
Flags
-
--app-id -
Filter by appID
-
--name -
Filter by name
Output formatting flags
-
--allow-missing-template-keys -
If true, ignore any errors in templates when a field or map key is missing in the template. Only applies to golang and jsonpath output formats.
-
-o,--output -
Output format. One of: (json, jsonpath, jsonpath-as-json, jsonpath-file).
-
--template -
Template string or path to template file to use when –output=jsonpath, –output=jsonpath-file.
algolia crawler pause
algolia crawler pause <crawler_id>... [flags]
Pause one or multiple crawlers.
Examples
1
2
3
4
5
6
# Pause the crawler with the ID "my-crawler"
$ algolia crawler pause my-crawler
# Pause the crawlers with the IDs "my-crawler-1" and "my-crawler-2"
$ algolia crawler pause my-crawler-1 my-crawler-2
algolia crawler reindex
algolia crawler reindex <crawler_id>... [flags]
Reindex one or multiple crawlers.
Examples
1
2
3
4
5
6
# Reindex the crawler with the ID "my-crawler"
$ algolia crawler reindex my-crawler
# Reindex the crawlers with the IDs "my-crawler-1" and "my-crawler-2"
$ algolia crawler reindex my-crawler-1 my-crawler-2
algolia crawler run
algolia crawler run <crawler_id> [flags]
Run a crawler.
Examples
1
2
3
# Run the crawler with the ID "my-crawler"
$ algolia crawler run my-crawler
algolia crawler stats
algolia crawler stats <crawler_id> [flags]
Get statistics about a crawler.
Examples
1
2
3
# Get statistics about the crawler with the ID "my-crawler"
$ algolia crawler stats my-crawler
Output formatting flags
-
--allow-missing-template-keys -
If true, ignore any errors in templates when a field or map key is missing in the template. Only applies to golang and jsonpath output formats.
-
-o,--output -
Output format. One of: (json, jsonpath, jsonpath-as-json, jsonpath-file).
-
--template -
Template string or path to template file to use when –output=jsonpath, –output=jsonpath-file.
algolia crawler test
algolia crawler test <crawler_id> --url <url> [-F <file>] [flags]
Test a URL on a crawler.
Examples
1
2
3
4
5
6
# Test the URL "https://www.example.com" against the crawler with the ID "my-crawler"
$ algolia crawler test my-crawler --url https://www.example.com
# Test the URL "https://www.example.com" against the crawler with the ID "my-crawler" and override the configuration with the file "config.json"
$ algolia crawler test my-crawler --url https://www.example.com -F config.json
Flags
-
-F,--config -
The configuration file to use to override the crawler’s configuration. (use “-“ to read from standard input)
-
-u,--url -
The URL to test.
algolia crawler unblock
algolia crawler unblock <crawler_id> [flags]
Unblock a crawler.
Examples
1
2
3
# Unblock the crawler with the ID "my-crawler"
$ algolia crawler unblock my-crawler
