Guides / Sending and managing data / Send and update your data

You can add transformations to your connector’s destination to manipulate data before it’s written to your Algolia indices.

To use transformations, create a task that specifies both the data source and the destination (usually an Algolia index). You can add a transformation when creating the task or directly to an existing destination.

To learn more about connectors, see Connectors overview.

Transformations are written as small JavaScript functions.

Only one transformation can be added to a destination.

Get started

  1. Go to the Algolia dashboard and select your Algolia application.
  2. On the left sidebar, select Data sources.
  3. Open the Connectors page.
  4. Select a connector and click Connect to start creating your task.
  5. Configure your data source: create a new source or select an existing one.
  6. To create your transformation function, click Create transformation. When writing a transformation, you have these choices:

    • Write your own functions directly in the code editor, which includes autocompletion and syntax highlighting.
    • Use predefined helper functions for common Algolia-specific tasks. To use a helper function, click Helper functions and select from the list. For more information, see Helper functions.
  7. Configure your destination: create a new destination or select an existing one.
  8. Configure when your task should run and specify the operation type for writing your Algolia index records.
  9. Confirm whether you want to run your task immediately. To see your transformation in action, click Run and inspect your index records after the task completes.

Helper functions

The following helper transformations are available to use, or extend, in your own transformations.

Helper type Description
Reshape Includes functions for adding, or removing attributes from a record.
Filters Exclude records based on a condition.
Convert Change the data type of attributes, trim or normalize them, and convert data types to a specific ISO format.
Compute New properties: calculate a discounted price and show as a new discount attribute.
Ranking Optimize custom ranking by prioritizing recent activity and adjusting attribute importance.
   
Query Categorization Manage hierarchical categories by converting a flat list into a hierarchical structure.
Fetch / API Enrich your Algolia records with data from external APIs.

Preview transformation results

When you create a transformation, you can preview the function on the right side of the code editor.

Preview your transformation running on a sample record

Preview your transformation on a sample record.

The preview includes the following information:

  • Input Record. A sample record, taken from the data source before applying the transformation.

  • Transformation Preview. Click Try it out to see the results of your transformation on the sample record.

Not all connectors support fetching sample records.

Debug transformations

Use the Connector Debugger for insights into the status of your tasks, including transformations.

Observe the status of previous executions of your transformations

Inspect the status of previous runs of your transformations.

Error handling

If your transformation function throws an error, it may fail the task depending on the task type and setup.

Take care to avoid errors when writing transformation functions. Use try and catch blocks to handle errors gracefully.

1
2
3
4
5
6
7
8
9
async function transform(record, helper){
  try {
    // Your transformation code here.
  } catch (error) {
    // Return the original record to avoid breaking the task,
    //  or return undefined to delete and ignore the record.
    return record;
  }
}

You should also make allowances for when your transformation function doesn’t work as expected. For example, use optional chaining to access missing properties.

1
2
3
4
5
6
async function transform(record, helper){
  // Accessing a property that may not exist.
  const value = record?.property?.nestedProperty;

  // Your transformation code here.
}

Broken transformations and consecutive failures

Algolia will stop your transformation if it encounters multiple consecutive failures within a short period.

The transformation will restart if its code changes or after a specific time.

To avoid this, write transformation functions to avoid errors and preview the transformation before running it.

Performance considerations

Adding a transformation adds an overhead to the ingestion process. Try to keep the JavaScript transformation as simple as possible.

Indexing strategy impact

Data transformations modify your records before indexing. By default, the entire record is returned after transformation. However, the correct approach depends on your indexing strategy you choose.

The following example adds a new property to the record and returns it. This is the default behavior.

1
2
3
4
async function transform(record, helper){
  record.newProperty = "foo"
  return record;
}

If your data is updated from multiple sources (API clients, Connectors, Ecommerce integrations) you need to carefully choose the indexing strategy.

Sometimes, returning the whole record can lead to data loss or other unwanted behavior. The following sections explain the different synchronization strategies and their different output behaviors.

Full reindexing and full record updates

These methods completely replace existing records. Therefore, your transformation must return the entire modified record, including any additions, deletions, or changes.

Modify existing or new record:

1
2
3
4
5
6
7
8
9
10
11
12
13
async function transform(record, helper){
  // Add a property.
  record.newProperty = "foo";

  // Remove a property.
  delete record.existingProperty;

  // Modify a property.
  record.existingProperty2 = "a new value"

  // Return the full modified record.
  return record;
}

Delete record:

1
2
3
async function transform(record, helper) {
  return undefined;
}

Create new records:

1
2
3
4
5
6
7
8
9
10
11
12
async function transform(record, helper){
  // Create a brand new record. You have to set the objectID yourself!
  newRecord = {};
  newRecord.objectID = "my-new-object";

  // Create a new object, from the current record. You have to change the objectID!
  copyRecord = { ...record };
  copyRecord.objectID = record.objectID + "-copy";

  // Return the initial record, the copy, and the brand new
  return [record, newRecord, copyRecord];
}

Partial record updates

This method only updates specific attributes. Your transformation should only return the object ID and the modified or added attributes. Existing attributes not included in the returned object remain unchanged. You cannot delete records with partial updates.

Modify an existing record:

1
2
3
4
5
6
7
8
9
10
async function transform(record, helper){
  // We only keep the objectID
  let partialRecord = { objectID: record.objectID };

  // We only add or modify what we need.
  partialRecord.newProperty = "foo";
  partialRecord.existingProperty = "new value";

  return partialRecord;
}

Add new records:

1
2
3
4
5
6
7
8
async function transform(record, helper){
  // Create a brand new record. You have to set yourself the objectID!
  newRecord = {};
  newRecord.objectID = "my-new-object";

  // Return the new record. The original record will be untouched.
  return [newRecord];
}
Did you find this page helpful?