Troubleshooting monitoring issues
Monitoring issues refer to problems that arise after a site has been crawled and its data sent to Algolia. You might notice discrepancies in the number of records from one crawl to the next or may wish to refresh your site’s content between scheduled crawls while aiming to sidestep potential complications.
Difference in the number of records between crawls
Problem
A noticeable increase or decrease in the number of records between crawls may be due to:
- Temporary site problems may prevent the crawler from finding everything.
-
Changes to the site. If your site changed but the crawler configuration wasn’t, the crawl might not fully capture the site content.
For example, if your company’s job listings page moves from
www.example-company.com/careers
towww.work-at-example-company.com
, the crawl wouldn’t include the new job listing URLs. - Timing issues with site updates and crawls. For example, you update your site’s content at 8 AM every day, but the crawler schedule is 9 AM. If the updates aren’t completed on time, the crawl may miss the new content, and users might come across outdated information.
Solutions
Issue | Solution |
---|---|
Temporary site problems | Perform a manual re-crawl once the site issues are resolved |
Changes to the site | Update your crawler configuration as needed and perform a re-crawl. For large changes, consider updating the safety check threshold |
Timing issues | Allow ample time for updates to complete before the crawler schedule |
Update between scheduled crawls
Problem
Your crawl might be scheduled to run at specific intervals, but there could be occasions when immediate updates are necessary. For example, a news site might need to publish breaking news immediately.
Solution
To add just a few pages, adjust the crawler configuration and then manually re-crawl from the Crawler dashboard.