Skip to main content

Documentation Index

Fetch the complete documentation index at: https://algolia.com/llms.txt

Use this file to discover all available pages before exploring further.

  • Type: boolean
  • Default: false
By default, the Crawler doesn’t extract links from pages with the robots meta tag set to nofollow or none. It also ignores links with the rel attribute set to nofollow. If ignoreNoFollowTo is true, the Crawler ignores any nofollow directives.
Links that don’t match any pattern in your configuration are always ignored.

​
Examples

JavaScript
{
  ignoreNoFollowTo: true,
}
Last modified on March 23, 2026