> ## Documentation Index
> Fetch the complete documentation index at: https://algolia.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Suggestions

> How to interpret the Crawler's suggestions for configuration improvements.

After each site crawl, the overview page provides "Suggestions" to help optimize your crawler configuration.

To access these suggestions, click **View in Suggestions** on the Crawler's overview page or **Suggestions** in the sidebar menu.

<img src="https://mintcdn.com/algolia/KSYHF7soFPXylOAb/doc/tools/crawler/getting-started/suggestions.png?fit=max&auto=format&n=KSYHF7soFPXylOAb&q=85&s=6d1cb61d11428e1ecb155ab0edd42a94" alt="Example list of crawler configuration improvement suggestions" width="1600" height="757" data-path="doc/tools/crawler/getting-started/suggestions.png" />

Click **Snooze** to hide a suggestion for 30 days.
Click **Dismiss** to permanently hide a suggestion.

For information about potential errors,
see [Error messages](/doc/tools/crawler/troubleshooting/error-messages).

## Potential suggestions

| Suggestion                                                                  | Solution                                                                                                                                                                                             |
| --------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Automatic crawls schedule isn't set                                         | To ensure your data is always up-to-date, [schedule automatic crawls](/doc/tools/crawler/getting-started/crawler-configuration-visual#configuration-settings)                                        |
| Robots.txt file is missing or Algolia Crawler is disallowed                 | The crawler encountered issues with your site's [robots.txt file](/doc/tools/crawler/getting-started/create-crawler#robots-txt). Either the file is missing, or it doesn't allow the Algolia Crawler |
| Sitemap not found                                                           | Ensure efficient crawling by adding [`sitemaps`](/doc/tools/crawler/apis/configuration/sitemaps) to the crawler configuration                                                                        |
| Some HTML pages or documents are too big                                    | [Records exceed the maximum for your Algolia plan](/doc/tools/crawler/troubleshooting/indexing-issues#records-exceed-the-maximum-for-your-algolia-plan)                                              |
| URLs ignored or failed                                                      | [Review the crawl status](/doc/tools/crawler/troubleshooting/crawl-status)                                                                                                                           |
| Website HTML contains `<meta name="robots" content="NOINDEX,NOFOLLOW">` tag | [Remove these meta tags from the pages or ignore them in the crawler configuration](/doc/tools/crawler/troubleshooting/fetching-issues#meta-tags)                                                    |