> ## Documentation Index
> Fetch the complete documentation index at: https://algolia.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Large indexing jobs

> Algolia can handle a high load of indexing requests, but there are caveats to keep in mind.

export const Records = () => <Tooltip tip="A record is a searchable object in an Algolia index. Each record consists of named attributes." cta="Algolia records" href="/doc/guides/sending-and-managing-data/prepare-your-data#algolia-records">
    records
  </Tooltip>;

export const Index = () => <Tooltip tip="An Algolia index is a searchable dataset that consists of records and configuration settings. These settings define how the records are searched and ranked.">
    index
  </Tooltip>;

Algolia can handle a large number of indexing requests.
But when you need to import tens or hundreds of millions of <Records /> in a short period of time,
be aware of the caveats.

<Note>
  Algolia always prioritizes search operations,
  so that indexing operations don't impact the search performance.
</Note>

## Preparation

If possible, keep Algolia in the loop when you plan to index a large number of records.
This lets Algolia monitor and optimize the configuration of your servers and indices,
for example, by fine-tuning the [sharding](https://en.wikipedia.org/wiki/Shard_\(database_architecture\)) of internal indices for your specific data.

Contact your customer success manager or the [Algolia support](https://support.algolia.com/hc/en-us/requests/new) team.

### Configure indices before uploading records

Configure an <Index /> **before** uploading records.
Setting the [`searchableAttributes`](/doc/api-reference/api-parameters/searchableAttributes) parameter beforehand is particularly important
to ensure the best indexing speed.
By default, Algolia indexes all attributes,
but you'll likely want to search in only a few of them.
For more information, see [Searchable attributes](/doc/guides/managing-results/must-do/searchable-attributes).

### Ensure the data fits on your servers

For best performance, Algolia stores all indices in memory on your servers.
It's best to keep the combined size of your indices below 80% of the total allocated RAM.
When the index size exceeds the RAM capacity,
the solid-state hard drive is used,
which is much slower.

Your index is usually between **two and three times larger** than the raw size of your data.
The exact factor depends on the structure of your data and your index configuration.

## Data upload

### Batch indexing jobs

Algolia's [API clients](/doc/libraries) have `saveObjects` helper methods
for uploading records in batches.
This is more efficient than uploading one record after the other.
Batches between 1,000 and 100,000 records tend to be optimal,
depending on the average record size.
Each batch should be smaller than 10 MB.
The API can handle batches of up to 1 GB,
but smaller batches lead to faster indexing.

### Multi-thread your indexing

You can use several parallel workers to make multiple indexing requests in parallel.

### Large datasets

If you have large indexing jobs, you might run into limitations
which you can avoid by optimizing the settings of the API clients:

1. Adjust the batch size when using the `saveObjects` method. The optimal size depends on your average record size, so you may need to experiment with different values to find what works best.
2. Compress the record.

You should also inspect the Algolia HTTP error to decide whether it's too big to process.