Skip to main content
Use the Monitoring tool to examine a crawler’s status. URLs can have one of three crawl statuses:
  • Success
  • Ignored (skipped) URLs didn’t match any of your configuration’s pathsToMatch or they did match one of your exclusionPatterns
  • Failed Each failed URL fits into one of these categories:
    • Fetching. URLs that failed when accessing your site
    • Extraction. URLs that failed when attempting to extract data from your site.
    • Indexing. URLs that weren’t sent to Algolia.
  • Internal An internal crawler error

​
Failed URL details

To get more details about a failed URL:
  1. Open the URL inspector
  2. Click Test URL to open the URL Tester.
  3. Click Run Test.
In the following example, the URL failed because the record was too big: Screenshot of the URL Tester showing a failed request with an error: 'Record extracted is too big (11KB), the maximum is 9.765625KB.' For more information, see:
Last modified on June 19, 2026