> ## Documentation Index
> Fetch the complete documentation index at: https://www.algolia.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Recover from ingestion failures

> Find out why an import or connector run failed, fix the cause, turn a task back on, and run again without duplicating records.

An ingestion failure is any import or update that doesn't end with the records (A record is a searchable object in an Algolia index. Each record consists of named attributes.) you expected in your index (An Algolia index is a searchable dataset that consists of records and configuration settings. These settings define how the records are searched and ranked.).
Where the failure shows up, and what the index looks like afterwards, depends on how you send data:

| Route | Where the failure appears |
| - | - |
| [API client](#api-client) | The method throws an error. The API client doesn't report the failure again later. |
| [Connector task](#connector-task) | The run shows a failed outcome in the [Connector Debugger](https://dashboard.algolia.com/connectors/debugger). After five consecutive failures, Algolia turns the task off and emails you. |
| [Push to Algolia connector](#push-to-algolia-connector) | Check the push's event in the Connector Debugger or through the [Ingestion API](https://www.algolia.com/doc/libraries/sdk/methods/ingestion). You can also configure the push request to wait for the processing result. |

What a failure leaves in your index depends on the update strategy.
For more information, see [Index contents after a failed run](#index-contents-after-a-failed-run).

## Identify ingestion failures

To find the error, follow the instructions for the process you used to send data:
API client, connector task, or Push to Algolia connector.

### API client

API clients report failures when they happen.
If Algolia rejects a request, check the HTTP status code and error message.
For example, an oversized record can cause an HTTP `400` error with a message that identifies the record and its position in the batch:

```txt Error message theme={"system"}
Record at the position 400 objectID=too-big is too big size=2000051/100000 bytes.
```

By default, [`saveObjects`](https://www.algolia.com/doc/libraries/sdk/methods/search/save-objects) sends records in batches of 1,000.
You can configure the batch size.

If Algolia rejects a batch, the method throws an error.
Algolia doesn't roll back batches it accepted before the failure.
It may still be indexing them.
To confirm that indexing has finished before you continue,
[wait for the indexing task](https://www.algolia.com/doc/guides/sending-and-managing-data/send-and-update-your-data/in-depth/index-operations-are-asynchronous.md).

### Connector task

Open the [Connector Debugger](https://dashboard.algolia.com/connectors/debugger).
In the **Runs** list, check the **Outcome** column.
A red dot indicates that the run failed.
Click the failed run's row for details.

Check the run's progress.
Look for events with a **Critical** status and read their error messages.

Use the events to identify whether the connector couldn't read the source or Algolia rejected the records.
Check the **Critical** event's type to identify the cause:

* **Critical event of type Fetch, after a series of Started and Retried events.**
  The connector couldn't read the source.
  The run tries to fetch the source up to five times before it fails, waiting about a minute between attempts,
  so a run against an unreachable source fails after about five minutes.
  The error names the cause, for example `[URL Fetcher] response failed with status code: 404`.
* **Record event with Critical status.**
  The connector read the source and sent records to Algolia in batches, and Algolia rejected one batch.
  Each accepted batch is a **Record** event with the status **Succeeded**.
  The rejected batch's error message includes the position of the record that caused it.
  Algolia doesn't retry rejected batches.

![Screenshot of connector events showing a successful record batch and a critical record event with its error message.](https://mintcdn.com/algolia/xW6LWIEba82zQ_ns/doc/guides/sending-and-managing-data/send-and-update-your-data/how-to/connector_events.png?fit=max\&auto=format\&n=xW6LWIEba82zQ_ns\&q=85\&s=233a608adb361140c07fb6d8bf9a15c6)

#### Ingestion API

To monitor connector runs in your code, use the Ingestion API:

* [`listRuns`](https://www.algolia.com/doc/libraries/sdk/methods/ingestion/list-runs) returns each run's status, outcome, and a reason.
* [`getRun`](https://www.algolia.com/doc/libraries/sdk/methods/ingestion/get-run) returns details for one run, including the progress counters `expectedNbOfEvents` and `receivedNbOfEvents`.
* [`listEvents`](https://www.algolia.com/doc/libraries/sdk/methods/ingestion/list-events) returns the events with their messages.
* [`listTasks`](https://www.algolia.com/doc/libraries/sdk/methods/ingestion/list-tasks) shows whether each task is `enabled`.

### Push to Algolia connector

By default, a push request returns `OK` with a `runID` and an `eventID` without waiting for Algolia to finish processing the records.

A Push to Algolia task uses the same run for multiple push requests.
The Connector Debugger shows this run as **Listening**.
The run doesn't report an outcome for each push.

To wait for the processing result,
set the `watch` parameter of [`push`](https://www.algolia.com/doc/libraries/sdk/methods/ingestion/push) or [`pushTask`](https://www.algolia.com/doc/libraries/sdk/methods/ingestion/push-task) to `true`.
The request waits for the operation to complete, so you don't need to poll `getEvent` yourself.

* If Algolia rejects the records, the request fails with an API error.

  If you call the [REST endpoint](https://www.algolia.com/doc/rest-api/ingestion/push-task.md) directly with `watch` set to `true`,
  Algolia returns HTTP `400` when it rejects a batch.
  The response includes the run's events.
  A `record` event with the status `critical` identifies the rejected batch.

* If the request succeeds, Algolia has finished processing the push.

You can also inspect an individual push event.
Call [`getEvent`](https://www.algolia.com/doc/libraries/sdk/methods/ingestion/get-event) with the `runID` and `eventID` from the push response.
A direct `getEvent` call can return HTTP `404` during the first few seconds because the event isn't available yet.

* If Algolia rejects the batch, the event has the type `record` and the status `critical`.
  Its message contains the API error.
* If Algolia accepts the batch, the event has the type `log` and the message `push request completed`.

For successful batches, the run also contains separate `record` events with the status `succeeded`.
Call `listEvents` with the `runID` to inspect these events.

The same events appear in the Connector Debugger under the task's run.

## Fix ingestion errors

| Error | Cause | What to do |
| - | - | - |
| `Record at the position N objectID=... is too big` | A record exceeds your plan's [record size limit](https://www.algolia.com/doc/guides/scaling/algolia-service-limits.md#application-record-and-index-limits). Algolia rejects the whole batch. | Shrink the record, for example by [reducing its size](https://www.algolia.com/doc/guides/sending-and-managing-data/prepare-your-data/how-to/reducing-object-size.md) or [splitting it](https://www.algolia.com/doc/guides/sending-and-managing-data/prepare-your-data/how-to/indexing-long-documents.md), or drop it with a [transformation](https://www.algolia.com/doc/guides/sending-and-managing-data/send-and-update-your-data/how-to/transform-your-data-with-code.md). |
| `[URL Fetcher] response failed with status code: 404` or another HTTP error | The connector's source URL no longer works. The dashboard checks the source URL when you save it. The source can stop working after this check. | Fix the file or URL, then edit the source if the URL changed. |
| `Method not allowed with this API key` | The API key lacks a permission, for example `deleteIndex` for [`deleteBy`](https://www.algolia.com/doc/libraries/sdk/methods/search/delete-by) or for cleaning up after [`replaceAllObjects`](https://www.algolia.com/doc/libraries/sdk/methods/search/replace-all-objects). | Add the permission or use another key. For more information, see [API key permissions](https://www.algolia.com/doc/guides/security/api-keys#access-control-list-acl). |
| `Index not allowed with this API key` | The API key's [index restrictions](https://www.algolia.com/doc/guides/security/api-keys/in-depth/api-key-restrictions.md#indices) exclude the target index, often the temporary index for a full reindex. | Widen the key's index restriction, for example to `INDEX_NAME*`. |
| `Task ... is disabled` | The task failed five times in a row and Algolia turned it off. | [Turn the task back on](#turn-a-task-back-on) after fixing the cause. |

Check that each record has the correct `objectID`.
An incorrect `objectID` can create or update the wrong record without returning an error.

## Turn a task back on

After five consecutive failed runs, Algolia turns the connector task off and sends an email to the team members of that Algolia application.

In the Connectors **Tasks** list, the task's status icon turns red, and **Run** returns `Task ... is disabled`.

To turn a task back on:

1. Fix the cause.
   Running a task whose source is still broken counts as another failure.
2. In the **Tasks** list, open the task's **⋮** menu and select **Enable task**.
3. Select **Run**, or wait for the next scheduled run.
   Enabling a task doesn't run it.

## Run again

If you send a record again with the same `objectID`,
Algolia updates the existing record instead of creating a duplicate.

> [!NOTE]
>
> Before retrying partial updates, check whether repeating an operation could change the record again.
> For example, retrying an [`Increment`](https://www.algolia.com/doc/libraries/sdk/methods/search/partial-update-object) operation can increment the value twice.

### Index contents after a failed run

* **Full record updates and partial record updates** (API: [`saveObjects`](https://www.algolia.com/doc/libraries/sdk/methods/search/save-objects), [`partialUpdateObjects`](https://www.algolia.com/doc/libraries/sdk/methods/search/partial-update-objects)) write to your index as they go.
  Algolia keeps the records it indexed before the failure.
  Algolia doesn't index any records in the rejected batch.

  * For full record updates, running again updates existing records and adds missing records.
  * For partial updates, check the operations and the `createIfNotExists` setting before retrying.
* **Full reindexing** (API: [`replaceAllObjects`](https://www.algolia.com/doc/libraries/sdk/methods/search/replace-all-objects)) writes to a temporary index and replaces your index only after Algolia has accepted every batch.
  A failed run leaves your index unchanged and searchable.
  Running again starts over.
  * Connector tasks delete the temporary index, `INDEX_NAME.RUN_ID.tmp`, when the run fails.
  * `replaceAllObjects` deletes its temporary index, `INDEX_NAME_tmp_*`, only if your API key has the `deleteIndex` permission.
    Otherwise, delete it yourself.
* **Push** rejects the batch that contains a bad record.
  Push the batch again after fixing or removing the record.

### Records missing from your source

Running again doesn't remove records whose source items no longer exist, unless you use full reindexing.
For more information, see [Remove records missing from your source](https://www.algolia.com/doc/guides/sending-and-managing-data/send-and-update-your-data/in-depth/the-different-synchronization-strategies.md#remove-records-missing-from-your-source).

> [!NOTE]
>
> The [Shopify, BigCommerce, and commercetools integrations](https://www.algolia.com/doc/integration.md) handle deletions separately from import reruns.
> When a source platform reports a deletion,
> its integration removes the corresponding Algolia record without a full reindex.

## Verify the recovery

* **Connector task.** In the Connector Debugger, check that the new run's **Outcome** column indicates success (a green dot).
* **API client.** The method returned without throwing an error, and the [indexing tasks have finished](https://www.algolia.com/doc/guides/sending-and-managing-data/send-and-update-your-data/in-depth/index-operations-are-asynchronous.md).
* **Push.** If you set `watch` to `true` on the recovery request,
  confirm that it completed without an error.
  Otherwise, inspect that request's event and
  confirm that it has the type `log` and the message `push request completed`.

Compare the index with the expected output of your ingestion configuration.
Check the record count and retrieve representative records by `objectID` to confirm their values.
If records are present but a search doesn't return them, see [Troubleshooting search](https://www.algolia.com/doc/guides/managing-results/troubleshooting/troubleshooting-search.md).

## See also

* [Connectors](https://www.algolia.com/doc/guides/sending-and-managing-data/send-and-update-your-data/connectors.md)
* [Data synchronization strategies](https://www.algolia.com/doc/guides/sending-and-managing-data/send-and-update-your-data/in-depth/the-different-synchronization-strategies.md)
