Suggestions

After each site crawl, the overview page shows suggestions to help improve your crawler configuration. To access these suggestions, click View in Suggestions on the Crawler’s overview page. You can also click Suggestions in the sidebar menu.

Example list of crawler configuration improvement suggestions

Click Snooze to hide a suggestion for 30 days. Click Dismiss to permanently hide a suggestion. For information about potential errors, see Error messages.

Potential suggestions

Suggestion	Solution
Automatic crawls schedule isn’t set	To ensure your data is always up-to-date, schedule automatic crawls
Robots.txt file is missing or Algolia Crawler is disallowed	The crawler encountered issues with your site’s robots.txt file. Either the file is missing, or it doesn’t allow the Algolia Crawler
Sitemap not found	Ensure efficient crawling by adding `sitemaps` to the crawler configuration
Some HTML pages or documents are too big	Records exceed the maximum for your Algolia plan
URLs ignored or failed	Review the crawl status
Website HTML contains `<meta name="robots" content="NOINDEX,NOFOLLOW">` tag	Remove these meta tags from the pages or ignore them in the crawler configuration

Last modified on September 16, 2026

Get started

Extract data

Enrich data

Troubleshooting

APIs

Netlify plugin

Potential suggestions

​Potential suggestions

Potential suggestions