Tools
/
Crawler
/
Getting started
Dec 11, 2024
Suggestions
After each site crawl, the overview page provides “Suggestions” to help optimize your crawler configuration.
To access these suggestions, click View in Suggestions on the Crawler’s overview page or Suggestions in the sidebar menu.
Click Snooze to hide a suggestion for 30 days. Click Dismiss to permanently hide a suggestion.
Guide
Crawler error messages
Potential suggestions
| Suggestion | Solution |
|---|---|
| Automatic crawls schedule is not set | To ensure your data is always up-to-date, schedule automatic crawls |
| Robots.txt file is missing or Algolia Crawler is disallowed | The crawler encountered issues with your site’s robots.txt file. Either the file is missing, or it doesn’t allow the Algolia Crawler |
| Sitemap not found | Ensure efficient crawling by adding sitemaps to the crawler configuration |
| Some HTML pages or documents are too big | Records exceed the maximum for your Algolia plan |
| URLs ignored or failed | Review the crawl status |
Website HTML contains <meta name="robots" content="NOINDEX,NOFOLLOW"> tag |
Remove these meta tags from the pages or ignore them in the crawler configuration |
Did you find this page helpful?
