Guides / Sending and managing data / Send and update your data

Sending Records in Batches

Whether you’re using the API or Dashboard, it’s best to send several records at a time instead of pushing them one by one. This has many benefits: it reduces network calls and speeds up indexing. Customers with the largest number of records will see the biggest impact on performance, but we recommend everyone to send indexing operations in batches whenever possible.

For example, let’s say you’re fetching all data from your database and end up with a million records to index. That would be too big to send in one take. But sending one record at a time would take too long. You would get much faster indexing by splitting the whole lot into smaller chunks of records, and sending these one by one.

Example

Continuing with our example. You have a million records to index. Pushing them in a single call wouldn’t likely be an option, because Algolia limits you to 1 GB per request. Plus, sending that much data would fail anyway before ever reaching the API.

Your first instinct might be to loop over each record and send them with the addObjects method. The problem is that you would perform a million individual network calls, which is bad from a performance standpoint both on your end and on Algolia’s side.

A much leaner approach is to split your collection of records into smaller collections, then send each chunk one by one. For optimal indexing performance, we recommend a batch size of ~10 MB, which represents between 1,000 or 10,000 records depending on the average record size.

Batching records won’t reduce your operations count. Algolia counts indexing operations per record, not per method call, so batching records won’t be counted differently than indexing them one by one.

Using the API

To push records in batches, you need to chunk your records, then loop over each chunk and send it to Algolia with the addObjects method.

If you need to send data from large files and handle concurrency in JavaScript, you can also use algolia-cli with the algolia import command.

1
2
3
4
5
6
7
$client = new \AlgoliaSearch\Client('YourApplicationID', 'YourAdminAPIKey');
$index = $client->initIndex('actors');

$records = json_decode(file_get_contents('actors.json'), true);

// Batching is done automatically by the API client
$index->saveObjects($records, ['autoGenerateObjectIDIfNotExist' => true]);

With this approach, you would only make 100 API calls. Depending on the size of your records and your network speed, you could create bigger or smaller chunks.

For more information, see our Importing Data via the API tutorial.

Using the Dashboard

You can also send your records in your Algolia dashboard.

Add records manually

  • Go to your dashboard and select your index.
  • Click Manage current index then Add manually.
  • Copy/paste your chunk in the JSON editor, then click Push record.
  • Repeat for all your chunks.

Upload a file

  • Go to your dashboard and select your index.
  • Click Manage current index then Upload file.
  • Either click the file upload area to select the file where your chunk is, or drag and drop it on it.
  • Upload will start automatically.
  • Repeat for all your chunks.

For more information, see our Importing Data via the Dashboard tutorial.

Did you find this page helpful?