> ## Documentation Index > Fetch the complete documentation index at: https://algolia.com/llms.txt > Use this file to discover all available pages before exploring further. # Crawl URLs > Crawls the specified URLs, extracts records from them, and adds them to the index. If a crawl is running (the crawler's reindexing property is true), the records are added to a temporary index. This operation is rate-limited to 500 requests every 24 hours. ## OpenAPI ````yaml specs/crawler.yml post /1/crawlers/{id}/urls/crawl openapi: 3.1.0 info: title: Crawler API summary: The Crawler API lets you manage and run your crawlers description: > ## Base URL The base URL for making requests to the Crawler API is: - `https://crawler.algolia.com/api` **All requests must use HTTPS.** ## Availability and authentication To authenticate your API requests, use the **basic authentication** header: - `Authorization: Basic ` Where `` is a base64-encoded string `:`. - ``. The Crawler user ID. - ``. The Crawler API key. You can find both on the [Crawler settings](https://dashboard.algolia.com/crawler/settings) page in the Algolia dashboard. The Crawler credentials are different from your regular Algolia credentials. ## Request format Request bodies must be JSON objects. ## Parameters Parameters are passed as query parameters for GET requests, and in the request body for POST and PATCH requests. Query parameters must be [URL-encoded](https://developer.mozilla.org/en-US/docs/Glossary/Percent-encoding). Non-ASCII characters must be UTF-8 encoded. ## Response status and errors The Crawler API returns JSON responses. Since JSON doesn't guarantee any specific ordering, don't rely on the order of attributes in the API response. Successful responses return a `2xx` status. Client errors return a `4xx` status. Server errors are indicated by a `5xx` status. Error responses have a `message` property with more information. ## Version The current version of the Crawler API is version 1, indicated by the `/1/` in each endpoint's URL. version: 1.0.0 servers: - url: https://crawler.algolia.com/api description: The URL of the Crawler API. security: - BasicAuth: [] tags: - name: actions x-displayName: State description: > Change the state of crawlers, such as pausing crawl schedules or testing the crawler with specific URLs. - name: config x-displayName: Configuration description: > In the Crawler configuration, you specify which URLs to crawl, when to crawl, how to extract records from the crawl, and where to index the extracted records. The configuration is versioned, so you can always restore a previous version. It's easiest to make configuration changes on the [Crawler page](https://dashboard.algolia.com/crawler) in the Algolia dashboard. The editor has autocomplete and built-in validation so you can try your configuration changes before committing them. - name: crawlers x-displayName: Manage description: | A crawler is an object with a name and a configuration. Use these endpoints to create, rename, and delete crawlers. - name: domains x-displayName: Domains description: List registered domains. - name: tasks x-displayName: Tasks description: Task operations. paths: /1/crawlers/{id}/urls/crawl: post: tags: - actions summary: Crawl URLs description: > Crawls the specified URLs, extracts records from them, and adds them to the index. If a crawl is running (the crawler's `reindexing` property is `true`), the records are added to a temporary index. This operation is rate-limited to 500 requests every 24 hours. operationId: crawlUrls parameters: - $ref: '#/components/parameters/CrawlerIdParameter' requestBody: content: application/json: schema: title: crawlUrls type: object properties: urls: type: array description: URLs to crawl. items: type: string example: - https://www.algolia.com/products/crawler/ save: type: boolean description: > Whether the specified URLs should be added to the `extraURLs` property of the crawler configuration. If unspecified, the URLs are added to the `extraUrls` field only if they haven't been indexed during the last reindex. required: - urls responses: '200': $ref: '#/components/responses/ActionAcknowledged' '400': $ref: '#/components/responses/InvalidRequest' '401': $ref: '#/components/responses/MissingAuthorization' '403': $ref: '#/components/responses/NoRightsOnCrawler' components: parameters: CrawlerIdParameter: name: id in: path description: Crawler ID. required: true schema: $ref: '#/components/schemas/CrawlerID' responses: ActionAcknowledged: description: OK content: application/json: schema: title: actionAcknowledged type: object properties: taskId: $ref: '#/components/schemas/TaskID' required: - taskId InvalidRequest: description: Invalid request. content: application/json: schema: title: invalidRequest type: object properties: error: title: invalidRequestError type: object properties: code: type: string errors: type: array items: title: errorItem type: object properties: message: type: string code: type: string line: type: integer position: type: integer required: - message example: message: url is not defined line: 5 message: type: string example: code: malformed_id required: - error MissingAuthorization: description: Authorization information is missing or invalid. NoRightsOnCrawler: description: >- The user doesn't have enough rights on the specified Crawler, or it doesn't exists. schemas: CrawlerID: type: string description: Universally unique identifier (UUID) of the crawler. example: e0f6db8a-24f5-4092-83a4-1b2c6cb6d809 TaskID: type: string description: Universally unique identifier (UUID) of the task. example: 98458796-b7bb-4703-8b1b-785c1080b110 securitySchemes: BasicAuth: type: http scheme: basic ````