Tools / Crawler / Troubleshooting

Troubleshooting by crawl status

Crawler status messages inform you of successful or failed crawl attempts. Use the Monitoring tool to examine a crawler’s status. URLs can have one of three crawl statuses:

  • Success.
  • Ignored (skipped). URLs didn’t match any of your configuration’s pathsToMatch or did match one of your exclusionPatterns
  • Failed. Each failed URL fits into one of four categories:
    • Fetching. URLs that failed when accessing your site
    • Extraction. URLs that failed when attempting to extract data from your site.
    • Indexing. URLs that weren’t sent to Algolia.
  • Internal. An internal crawler error

Failed URL details

To get more details about a failed URL:

  1. Open the URL Inspector
  2. Click Test this URL to open the URL Tester.
  3. Click Run Test.

In the following example, the URL failed because the record was too big:

Error message in the URL Tester

Did you find this page helpful?