Skip to main content
With Recommend deduplication, you can refine your recommendations. It removes item variants from your AI model training. This offers several advantages:
  • Improves recommendation accuracy and speed by removing item variants before the training process.
  • Improves recommendation quality by using events from all item variants.
  • Separates deduplication from search so it doesn’t affect the search experience.

​
Recommend deduplication versus search deduplication

Algolia offers two types of deduplication:
  • Search deduplication removes duplicate items from search results. This can help improve the relevance of the results and make them more user-friendly.
  • Recommend deduplication removes duplicate items from Algolia AI recommendations. This can help improve the diversity of the recommendations and make them more personalized.
With Recommend deduplication, you train your AI models with all variants. The API then filters the recommendations to remove variants before sending you the results.

​
How Recommend deduplication works

Recommend deduplication adds two processes to your AI model training:
  • Pre-training process: generates a training dataset with only one variant per item. It also merges all events from variants of the same item.
  • Post-training process: restores the variants dropped during pre-training. They go back into the final set of recommendations.
Item variants share the same recommendations.

​
Set up the deduplication for a model

To deduplicate your recommendations, first declare an attribute for distinguishing variants. Then turn on deduplication when you configure a Recommend model. After that, verify that the recommendations are deduplicated.

​
Configure an attribute for distinguishing variants

First, choose which attribute defines records as variants:
  1. Go to the Algolia dashboard and select your Algolia .
  2. On the left sidebar, select Search.
  3. Select your Algolia index.
  4. On the Configuration tab, go to the Deduplication and Grouping page.
  5. In the Attribute for Distinct box, select or enter the attribute name you want to use to define variants. Set the attributeForDistinct in your index settings
Only use the distinct option if you also want to deduplicate search results.

​
Enable Recommend deduplication on your model

  1. Go to the Algolia dashboard and select your Algolia application.
  2. On the left sidebar, select Recommend.
  3. Create a new Recommend model, or edit an existing one, for the index where you set attributeForDistinct.
  4. In the Deduplicate recommendations section, turn on the Deduplicate recommendations switch. This shows the attribute you selected for defining variants.
  5. Continue to configure your Recommend model and click Save. Enable deduplication in your model training configuration

​
Verify the recommendations

To check that deduplication works for your recommendations, revisit the model configuration after training finishes:
  1. Go to the Preview section.
  2. Use the Search for a record box to search for an item that should have variants.
This displays the list of recommendations for the selected item. They shouldn’t contain any variants.

​
Examples

The following examples illustrate how Recommend deduplication works. The index has records for T-shirts in different colors and sizes:
  • One red T-shirt in one size (XS)
  • Two green T-shirts in two sizes (S, M)
  • Three blue T-shirts in three sizes (L, XL, XXL)
This example uses the Related items model to recommend the top 3 similar items, with and without deduplication. It configures the color attribute as attributeForDistinct.

​
Without deduplication

Without deduplication, recommendations include variants, such as blue (XXL) or green (M), except for the red T-shirt which doesn’t have any.

​
With deduplication

With deduplication, the recommendations don’t include any variants. This example dataset includes only three records, so it can generate only two recommendations. If you add a new item, such as an orange T-shirt, Recommend adds it as a third recommendation.
Last modified on September 16, 2026