Crawler: Schedule

About this parameter

Use schedule to choose an automatic crawling frequency that best suits your site’s needs.

Instead of manually starting a crawl each time, you can set up a schedule for automatic crawls. There are two ways to do this:

• Use the visual UI
• Add the schedule parameter to your configuration.

schedule uses Later.js syntax to specify when to crawl your website. Here are some key things to keep in mind when using Later.js syntax with the Crawler:

• The interval between two scheduled crawls must be at least 24 hours.
• To crawl daily, use “every 1 day” instead of “everyday” or “every day”.
• If you don’t specify a time, the crawl can happen anytime during the scheduled day.
• Specify times for the UTC (GMT+0) timezone
• Include minutes when specifying a time. For example, “at 3:00 pm” instead of “at 3pm”.
• Use “at 12:00 am” to specify midnight, not “at 00:00 am”.

Examples

1
2
3
{
  schedule: 'every 1 day at 3:00 pm',
}