01 Jun 2018

Synchronization

Introduction

After the initial import of the data, you’ll need to keep your index up-to-date with the latest additions and changes on your application or website.

You can update your records:

  • one by one in realtime or batch them
  • update a complete record or only a subset of its attributes
  • update a full index without generating inconsistencies on the search during the process

Incremental Updates

Once your initial data has been indexed, you will want to configure your application to sync your database changes with your indexes in Algolia.

Incremental updates are performed by referencing the objectID attribute of your records. By referencing this unique objectID, you can add, delete, and modify an existing record’s attributes.

Adding a new record (if one doesn’t exist)

See Add objects.

var objects = [{
  firstname: 'Jimmie',
  lastname: 'Carter',
  objectID: 'myID1'
}, {
  firstname: 'Warren',
  lastname: 'Speach'
}];

index.addObjects(objects, function(err, content) {
  console.log(content);
});

Note: the first record contains an objectID and the 2nd one doesn’t. In both cases the record is added to the index, but in the 2nd record Algolia generates a new objectID which you can retrieve in the response.

Updating existing records

See Update objects.

var objects = [{
  firstname: 'Jimmie',
  lastname: 'Carter',
  objectID: 'myID1'
}, {
  firstname: 'Warren',
  lastname: 'Speach',
  objectID: 'myID2'
}];

index.saveObjects(objects, function(err, content) {
  console.log(content);
});

Note: both records contain an objectID. If any record is missing an objectID, the method would fail without making any updates.

Partial updates

See Partial update objects.

In some complex use-cases, you may have several data sources having each a part of the data required to build your Algolia records.

When dealing with several sources to update a single object, you may face a versioning issue - like a laggy source trying to push an update that would be older than the one currently indexed and override it.

Algolia handles such conflicts as updates from several sources with Partial Updates. Partial updates only update a few attributes of your records without re-pushing the others.

Example:

  • You index an object
    { "name": "Benjamin", "objectID": 42 }
    
  • Source A send the “city” attribute with a partialUpdate
var objects = [{
  city: 'Paris',
  objectID: 42
}];

index.partialUpdateObject({objects, function(err, content) {
  console.log(content);
});
  • Once indexed, the updated object will be
    { "name": "Benjamin", "city": "Paris", "objectID": 42 }
    
  • Source B sends the “country” attribute with a partialUpdate
var objects = [{
  country: 'France',
  objectID: 42
}];

index.partialUpdateObject({objects, function(err, content) {
  console.log(content);
});
  • Once indexed, the object will be
    { "name": "Benjamin", "city": "Paris", "country": "France", "objectID": 42 }
    

Since the different sources are not dealing with the same attributes, they’ll never override something another source has pushed.

Deleting (or purging) records

In order to keep your data size as small as possible, we recommend purging records that are no longer necessary for search (for example, out of stock items).

Deleting one record using an objectID

See Delete objects.

index.deleteObjects(['myID1', 'myID2'], function(err, content) {
  if (err) throw err;

  console.log(content);
});

Deleting records using filters

See Delete by.

It’s possible to delete all records that match a certain query. For example, we might want to delete all records from years prior to 2017.

index.deleteBy({
    filters: 'year < 2017'
  },
  function(err) {
    if (!err) {
      console.log('success');
    }
  }
);

or we can delete all records within a certain geographic location.

index.deleteBy({
  filters: 'category:cars',
  aroundLatLng: '40.71, -74.01'
}, function(err, content) {
  if (err) throw err;

  console.log(content);
});

Calling this method with an empty query and no parameters will result in the index being cleared.

Atomic reindexing

In some cases, you may want to change the way your records are structured, and need to reindex all the data in an index.

In order to keep your existing service running while re-importing your data, we recommend the usage of a temporary index plus an atomical move.

A Move index operation will override all the settings of the destination index, except the replicas setting. For more information, check out our guide on How to perform an Atomic Reindex.

How Often to Send Updates

The appropriate frequency for sending Algolia updates depends on how often new data is added to your site and how quickly that data needs to be searchable.

For example, for an e-commerce shop:

  • You’ll want to update in real-time the price changes, or the availability of a product
  • You don’t necessarily need to update the number of sales that you use for the ranking, so you can send them by batches periodically every hour/day/week

You’ll need to find a balance between having the information in the search as fast as possible, and minimizing the number of operations (because it has an impact on the pricing / performance).

Try these tutorials

If you want to get started implementing an update strategy, we have a few tutorials you might find helpful:

© Algolia - Privacy Policy