When sending data to Algolia, it’s best to send several records simultaneously instead of individually. It reduces network calls and speeds up indexing, especially when you have a lot of records, but everyone should send indexing operations in batches whenever possible.
For example, you might decide to send all the data from your database and end up with a million records to index. That’s too big to send all at once because Algolia limits you to 1 GB per batch per request.
In reality, sending that much data in a single network call would fail before reaching the API. You could loop over each record and send them with the saveObjects method. The problem is that you would perform a million individual network calls, which would take way too long and saturate your Algolia cluster with indexing jobs.
A leaner approach is to split your collection of records into smaller collections, then send each chunk one by one. For optimal indexing performance, aim for a batch size of about 10 MB, representing between 1,000 and 10,000 records, depending on the average record size.
Batching records doesn’t reduce your operations count.Algolia counts indexing operations per record, not per method call, so from a pricing perspective, batching records is the same as indexing records individually.
Be careful when approaching your plan’s maximum number of records. If you’re close to the record limit,
batch operations may fail. The error message
“You have exceeded your Record quota” means the engine doesn’t know if the batch operation will update records or add new ones.
If this happens, upgrade to a plan with a higher record limit or reduce your batch size.
Using the API
When using the saveObjects method, the API client automatically chunks your records into batches of 1,000 objects.
If you want to upload large files, consider using the Algolia CLI with the algolia objects import command.
$client=new\AlgoliaSearch\Client('YourApplicationID','YourWriteAPIKey');$index=$client->initIndex('actors');$records=json_decode(file_get_contents('actors.json'),true);// Batching is done automatically by the API client$index->saveObjects($records,['autoGenerateObjectIDIfNotExist'=>true]);
1
2
3
4
5
6
7
8
9
10
require'json'require'algolia'client=Algolia::Search::Client.create('YourApplicationID','YourWriteAPIKey')index=client.init_index('actors')file=File.read('actors.json')records=JSON.parse(file)# The API client automatically batches your recordsindex.save_objects(records,{auto_generate_object_id_if_not_exist: true})
importjsonfromalgoliasearch.search_clientimportSearchClientclient=SearchClient.create('YourApplicationID','YourWriteAPIKey')index=client.init_index('actors')withopen('actors.json')asf:records=json.load(f)# Batching is done automatically by the API client
index.save_objects(records,{'autoGenerateObjectIDIfNotExist':True})
valclient=ClientSearch(ApplicationID("YourApplicationID"),APIKey("YourWriteAPIKey"))valindex=client.initIndex(IndexName("actors"))valstring=File("actors.json").readText()valactors=Json.plain.parse(JsonObjectSerializer.list,string)index.apply{actors.chunked(1000).map{saveObjects(it)}.wait()// Wait for all indexing operations to complete.
usingSystem;usingSystem.IO;usingSystem.Collections.Generic;usingAlgolia.Search.Clients;usingNewtonsoft.Json;publicclassActor{publicstringName{get;set;}publicstringObjectId{get;set;}publicintRating{get;set;}publicstringImagePath{get;set;}publicstringAlternativePath{get;set;}}publicclassAlgoliaIntegration{privateSearchClientclient;privateSearchIndexindex;publicAlgoliaIntegration(stringApplicationID,stringapiKey){client=newSearchClient("YourApplicationID","YourWriteAPIKey");index=client.InitIndex("actors");// Assuming the actors.json file is in the same directory as the executablestringjson=File.ReadAllText("actors.json");varsettings=newJsonSerializerSettings{ContractResolver=newNewtonsoft.Json.Serialization.CamelCasePropertyNamesContractResolver()};IEnumerable<Actor>actors=JsonConvert.DeserializeObject<IEnumerable<Actor>>(json,settings);// Batching/Chunking is done automatically by the API clientboolautoGenerateObjectIDIfNotExist=true;index.SaveObjects(actors,autoGenerateObjectIDIfNotExist);}}// To use the above class, you would instantiate it somewhere in your application startup logic// Example:// var algoliaIntegration = new AlgoliaIntegration("YourApplicationID", "YourWriteAPIKey");
importjava.io.FileInputStream;importjava.io.InputStream;importcom.fasterxml.jackson.databind.ObjectMapper;publicclassActor{// Getters/Setters ommittedprivateStringname;privateStringobjectId;privateintrating;privateStringimagePath;privateStringalternativePath;}// Synchronous versionSearchClientclient=DefaultSearchClient.create("YourApplicationID","YourWriteAPIKey");SearchIndex<Actor>index=client.initIndex("actors",Actor.class);ObjectMapperobjectMapper=Defaults.getObjectMapper();InputStreaminput=newFileInputStream("actors.json");Actor[]actors=objectMapper.readValue(input,Actor[].class);// Batching/Chuking is done automatically by the API clientbooleanautoGenerateObjectIDIfNotExist=true;index.saveObjects(Arrays.asList(actors),autoGenerateObjectIDIfNotExist);
packagemainimport("encoding/json""io/ioutil""github.com/algolia/algoliasearch-client-go/v3/algolia/search")typeActorstruct{Namestring`json:"name"`Ratingint`json:"rating"`ImagePathstring`json:"image_path"`AlternativeNamestring`json:"alternative_name"`ObjectIDstring`json:"objectID"`}funcmain(){client:=search.NewClient("YourApplicationID","YourWriteAPIKey")index:=client.InitIndex("actors")varactors[]Actordata,_:=ioutil.ReadFile("actors.json")_=json.Unmarshal(data,&actors)// Batching is done automatically by the API client_,_=index.SaveObjects(actors)}
With this approach, you would make 100 API calls instead of 1,000,000. Depending on your records’ sizes and your network speed, you could create bigger or smaller chunks.