> ## Documentation Index
> Fetch the complete documentation index at: https://algolia.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Algolia Crawler

> The Algolia Crawler extracts content from your site and makes it searchable.

Given a set of start URLs, the Crawler:

1. Visits these pages and extracts data that's relevant for search.
2. Discovers other pages through links (and extracts *their* data).
3. Sends your data to your [Algolia indices](/doc/guides/sending-and-managing-data/prepare-your-data/in-depth/prepare-data-in-depth#algolia-index). You can run the Crawler on a schedule to keep your Algolia indices up-to-date.

## Why you should use the Crawler

The Crawler simplifies uploading your data to Algolia and keeping your indices up-to-date.
Compared to using [the API clients or other methods](/doc/libraries) to index your data, using the Crawler has these benefits:

* You don't have to write and maintain code for extracting content, transforming it to indices, and scheduling periodic updates.
* It helps you extract data from unstructured content (such as HTML and PDF files).
* It can index your web pages when it's difficult to access the *sources*, for example, due to restricted access, or if you want to index different resources managed by different teams using different tools.

## Get started

To add, verify, and crawl your site, see [Create a new crawler](/doc/tools/crawler/getting-started/create-crawler).

If you want to use the Crawler to index a technical documentation site,
consider [DocSearch](https://docsearch.algolia.com),
which also comes with a search UI.
If you use Netlify to host your website,
use the [Netlify Crawler plugin](/doc/tools/crawler/netlify-plugin/quick-start).