Image classification and tagging

Image classification and tagging lets you extract meta information from product images. You might add search by image later, or not at all. Either way, enriching your ecommerce data through image classification can improve search and discovery.

You can’t store images directly in Algolia. Instead, store the image on a content delivery network (CDN) or web server and add the image URL to a field in your . When you retrieve a record from Algolia, use this URL to display the image in your app.

Why use image classification?

Retailers spend a lot of time building their catalogs. To offer relevant search and discovery, they often classify each item by hand, adding features like item type, material, or style. Removing some of this manual work lets you focus on what matters to your business, such as price, stock, and popularity. Visual recognition can pull this information automatically by analyzing each product image. It makes feature tagging more consistent. For example, you may have various names for the color “blue” within your product descriptions. Item descriptions could include “cerulean” or “sapphire,” but not “blue”. Without a consistent “blue” value, you might miss showing relevant products to users when they on or search for “blue” items. Image classification lets you add the “blue” tag consistently for all blue products. Image classification is especially valuable in consumer-to-consumer (C2C) marketplaces, where users may not describe products consistently or fully. Tags from image classification can increase the number of product attributes, making them more discoverable. Image classification helps beyond C2C marketplaces too. It’s useful anywhere your team tags features like “type,” “neckline,” and “sleeve length” by hand.

What does image classification and tagging entail?

This guide shows how to use a third-party API or platform to classify images. Use these classifications to enrich your Algolia records. It provides examples for Google Cloud Vision API and ViSenze. The process is the same for other providers, like Amazon Rekognition. The goal is to enrich your records so that each one includes more descriptive text. This text comes from running the product image through an image classifier, which returns classifications. Another way of thinking of classifications is “tags” or “labels.” Adding these classifications to your Algolia records helps surface them in searches, whether users search with text or images. Enriching your records with classifications is a two-step process:

Image classification - sending image URLs to a third-party image recognition platform to retrieve classifications.
Indexing - adding the relevant classification information to your Algolia records.

If you want to let users search by images, first enrich your records with the same image recognition platform you plan to use for search. If you skip this step, a user’s image won’t match records with the same or similar image.

Platform considerations

Google Cloud Vision API is an all-purpose image recognition API. It draws from a large set of image data, so it can give a wide range of accurate classifications. The downside is that its classifications aren’t specialized or well structured. All-purpose image recognition platforms can introduce irrelevant classifications. An image of a model wearing a t-shirt could return relevant classifications, like “t-shirt,” along with the shirt’s color and style. But it could also return classifications like “neck” and “arm,” if these appear in the image. Google Cloud Vision API returns tags and confidence scores of all objects that it identifies in an image. If a platform exists for your use case, for example ViSenze for fashion retail, use the specialized platform instead of a general one. Platforms built for a specific case usually produce better classifications. These platforms tailor their classifications to industry terms and structure them consistently. For example, ViSenze takes an image of a model wearing a t-shirt. It identifies only fashion-related objects, excluding things like “neck” and “arm.” For each item it includes, such as “t-shirt,” it returns attributes like “neckline,” “fit,” and “sleeve length,” with values like “v-neck,” “trim,” and “short.” You can be sure that all shirt images retrieve these same attributes in the same structure.

Before you begin

This tutorial requires a set of Algolia records, each with an image URL. You’ll also need access to an image recognition platform, such as Google Cloud Vision API. Algolia doesn’t search in your original data source, but in the data you index to Algolia. Algolia accepts and stores JSON data, meaning it doesn’t store image files. Instead, it’s common to index an image URL, so that you can display the image in your results.

JSON

{
  "title": "Men's Athletic Shirt",
  "objectID": "807281751",
  "imageURL": "https://images-na.ssl-images-amazon.com/images/I/41uIVaJOLdL.jpg"
}

This is a truncated example. Your records probably have other attributes, like price, stock, and sales rank.

Image classification

Image classification takes an image and returns a set of classifications or labels for it. As AI advances, image classification keeps improving and becomes more accessible to non-experts. With platforms like Google Cloud Vision API or ViSenze, you often just send an image URL and get classifications back in the response.

Using Google Vision API

If you haven’t already, create a Google Account and enable the Google Vision API for it. Set up authentication so that you can retrieve credentials and use the Vision API client library. The Google Vision API returns an array of classifications: JSON objects with different properties. Of these properties, description and score are the most useful. The score shows how certain the API is about the description. After initializing an instance of Google Cloud Vision’s Node.js client, you can write a function to retrieve labels from an image URL. The example below creates a getImageLabels function that takes a public image URL, the Algolia record’s objectID, and a scoreLimit. The scoreLimit is the threshold for how certain the platform must be about an object to include it in the classifications. Since score is a number between 0 and 1, the scoreLimit should be between 0 and 1 too. A higher scoreLimit means the API must be more certain about a label before it includes the label. You can write a function to retrieve just these or any other attributes you find useful. The getImageLabels example returns an object with a labels array. The array contains only label descriptions and scores, where scores were higher than the scoreLimit. The returned object also includes the original imageURL and objectID. The objectID is important for sending this data to your Algolia index later.

JavaScript

// Import the Google Cloud client libraries
const vision = require('@google-cloud/vision');

// Instantiate Google Vision client
const client = new vision.ImageAnnotatorClient();

// Retrieve labels
async function getImageLabels(imageURL, objectID, scoreLimit) {
  const [result] = await client.labelDetection(imageURL);
  const labels = result.labelAnnotations
    .filter((label) => label.score > scoreLimit)
    .map((label) => (
      {
        description: label.description,
        score: label.score,
      }
    ))
  return { imageURL, objectID, labels };
}

const classifiedImage = await getImageLabels("https://images-na.ssl-images-amazon.com/images/I/41uIVaJOLdL.jpg", "439784001", 0.5)

Result:

JavaScript

const classifiedImage = {
  imageURL: "https://images-na.ssl-images-amazon.com/images/I/41uIVaJOLdL.jpg",
  objectID: "439784001",
  labels: [
    {
      "description": "Outerwear",
      "score": 0.9513528,
    },
    {
      "description": "Azure",
      "score": 0.89286935,
    },
    {
      "description": "Sleeve",
      "score": 0.8724504,
    },
    {
      "description": "Bag",
      "score": 0.86443543,
    },
    {
      "description": "Grey",
      "score": 0.8404184,
    }
  ]
}

When fetching images from HTTP URLs, Google can’t guarantee that the request succeeds. Your request might fail if the host denies it, for example due to request throttling or denial-of-service prevention. It might also fail if Google throttles requests to the site to prevent abuse. Google advises against depending on externally hosted images for production apps.

Using ViSenze

When using a case specific platform like ViSenze, the general idea is the same. Setup an account and credentials, and send public image URLs to their Recognition API to receive classifications. You need to tailor your function to the data structure the platform returns. For example, the getImageLabels function below takes a public image URL, the Algolia record’s objectID, and a scoreLimit. The scoreLimit is the threshold for how certain the platform must be about an object to include it in the classifications. Since score is a number between 0 and 1, the scoreLimit should be between 0 and 1 too. A higher scoreLimit means the API must be more certain about a label before it includes the label. The function returns an object with an objects array. The objects array contains all identified objects, such as “t-shirt” or “belt,” along with their coordinates, labels, and scores. The returned object also includes the original imageURL and objectID. The objectID is important for sending this data to your Algolia index later.

JavaScript

// Retrieve labels
async function getImageLabels(imageURL, objectID, scoreLimit) {
  const formData = new FormData();

  formData.append("limit", "30");
  formData.append("tag_group", "fashion_attributes");
  formData.append("url", imageURL);

  return await fetch("https://virecognition.visenze.com/v1/image/recognize", {
    method: "POST",
    headers: {
      Authorization: "Basic YOUR_BASE64_ENCODED_VISENZE_KEY",
    },
    body: formData,
  })
    .then((res) => res.json())
    .then((res) => {
      if (res.status !== "OK" && res.error[0]) {
        console.log("handle ViSenze - recognition error", res.error[0]);
        return;
      }

      const classifiedImage = {
        imageURL,
        objectID,
        objects: [],
      };

      // `res.result[0].objects` contains the objects detected in the image
      res.result[0].objects.forEach((object, index) => {
        // Store coordinates of the current object
        classifiedImage.objects[index] = {
          x1: object.box[0],
          y1: object.box[1],
          x2: object.box[2],
          y2: object.box[3],
        };

        // Format categories, attributes and scores
        object.tags.forEach(({ tag, score }) => {
          const splittedTag = tag.split(":");
          score = parseFloat(score.toFixed(2));

          if (score > scoreLimit) {
            if (!(splittedTag[0] in classifiedImage.objects[index])) {
              classifiedImage.objects[index][splittedTag[0]] = [];
            }

            classifiedImage.objects[index][splittedTag[0]].push({
              label: splittedTag[1],
              score,
            });
          }
        });
      });

      return classifiedImage;
    }).catch((err) => console.error("Image classification error", err));
};

const classifiedImage = await getImageLabels("https://images-na.ssl-images-amazon.com/images/I/41uIVaJOLdL.jpg", "439784001", 0.5)

Result:

JavaScript

const classifiedImage = {
  imageURL: "https://images-na.ssl-images-amazon.com/images/I/41uIVaJOLdL.jpg",
  objectID: "439784001",
  objects: [
      {
        x1: 65,
        y1: 14,
        x2: 729,
        y2: 788,
        apparel: [{ label: "upper_body_garment", score: 0.98 }],
        product_color: [{ label: "blue", score: 0.91 }],
        neckline: [{ label: "round_neck", score: 0.81 }],
        // This is a truncated list of classifications.
        // Other classifications include category, closure_type, denim_wash_color, lower_body_garment, lower_body_length, outerwear, pants_fit_type, product_pattern, rise_type, sleeve_length, sleeve_style, upper_body_garment, upper_body_length
      },
  ],
}

Indexing image classifications

Once you’ve retrieved the classifications from your third-party image recognition platform, send them to Algolia. You can add classifications when you first index your data, or later with the browse method. browse lets you retrieve and update your data as needed.

Using Google Vision API

namespace Algolia;

using System;
using System.Collections.Generic;
using System.Net.Http;
using System.Text.Json;
using Algolia.Search.Clients;
using Algolia.Search.Http;
using Algolia.Search.Models.Search;

class SaveImageClassifications
{
  class Image
  {
    public required string ImageUrl { get; set; }
    public required string ObjectId { get; set; }
    public required List<Dictionary<string, object>> Objects { get; set; }
  }

  // Retrieve labels
  async Task<Image> GetImageLabels(string imageURL, string objectID, float scoreLimit)
  {
    // Implement your image classification logic here
    return await Task.Run(() =>
      new Image
      {
        ImageUrl = "",
        ObjectId = "",
        Objects = [],
      }
    );
  }

  async Task Main(string[] args)
  {
    try
    {
      // API key ACL should include editSettings / addObject
      var client = new SearchClient(new SearchConfig("ALGOLIA_APPLICATION_ID", "ALGOLIA_API_KEY"));

      var hits = await client.BrowseObjectsAsync<Image>(
        "INDEX_NAME",
        new BrowseParamsObject()
      );

      var records = hits.Select(hit => GetImageLabels(hit.ImageUrl, hit.ObjectId, 0.5f))
        .Select(src => src.Result)
        .ToList();

      // Update records with image classifications
      await client.PartialUpdateObjectsAsync("INDEX_NAME", records, true);
    }
    catch (Exception e)
    {
      Console.WriteLine(e.Message);
    }
  }
}

import 'package:algolia_client_search/algolia_client_search.dart';

final class Image {
  final String imageUrl;
  final String objectID;
  List<Map<String, dynamic>> objects;

  Image(String imageUrl, String objectID, List<Map<String, dynamic>> objects)
      : imageUrl = imageUrl,
        objectID = objectID,
        objects = objects;
}

Future<Map<String, dynamic>> getImageLabels(
    String imageURL, String objectID, double scoreLimit) async {
  // Implement your image classification logic here
  return {};
}

void saveImageClassifications() async {
  // API key ACL should include editSettings / addObject
  final client =
      SearchClient(appId: 'ALGOLIA_APPLICATION_ID', apiKey: 'ALGOLIA_API_KEY');

  final List<Map<String, dynamic>> records = [];

  BrowseResponse? browseResponse;
  do {
    final browseParams =
        BrowseParamsObject(hitsPerPage: 1000, cursor: browseResponse?.cursor);
    browseResponse = await client.browse(
        indexName: "INDEX_NAME", browseParams: browseParams);
    for (final hit in browseResponse.hits) {
      final props = hit.toJson();
      final imageWithLabels =
          await getImageLabels(props["imageUrl"], hit.objectID, 0.5);
      records.add(imageWithLabels);
    }
  } while (browseResponse.cursor != null);

  final batchParams = BatchWriteParams(
      requests: records
          .map((record) => BatchRequest(
                action: Action.partialUpdateObject,
                body: record,
              ))
          .toList());

  // Update records with image classifications
  await client.batch(
    indexName: "INDEX_NAME",
    batchWriteParams: batchParams,
  );
}

package main

import (
	"github.com/algolia/algoliasearch-client-go/v4/algolia/search"
)

func saveImageClassifications() {
	type Image struct {
		ImageURL string           `json:"imageURL"`
		ObjectID string           `json:"objectID"`
		Objects  []map[string]any `json:"objects"`
	}

	getImageLabels := func(imageURL, objectID string, _scoreLimit float64) Image {
		// Implement your image classification logic here
		return Image{
			ImageURL: imageURL,
			ObjectID: objectID,
			Objects:  []map[string]any{},
		}
	}

	client, err := search.NewClient("ALGOLIA_APPLICATION_ID", "ALGOLIA_API_KEY")
	if err != nil {
		// The client can fail to initialize if you pass an invalid parameter.
		panic(err)
	}

	images := []Image{}

	err = client.BrowseObjects("INDEX_NAME", search.BrowseParamsObject{}, search.WithAggregator(func(res any, err error) {
		if err != nil {
			panic(err)
		}

		browseRes, ok := res.(search.BrowseResponse)
		if !ok {
			panic("Invalid response")
		}

		for _, hit := range browseRes.Hits {
			props := hit.AdditionalProperties
			imageURL, _ := props["imageURL"].(string)
			images = append(images, getImageLabels(imageURL, hit.GetObjectID(), 0.5))
		}
	}))
	if err != nil {
		panic(err)
	}

	records := make([]map[string]any, len(images))
	for i, img := range images {
		records[i] = map[string]any{
			"imageURL": img.ImageURL,
			"objectID": img.ObjectID,
			"objects":  img.Objects,
		}
	}

	_, err = client.PartialUpdateObjects(
		"INDEX_NAME", records, search.WithCreateIfNotExists(true))
	if err != nil {
		panic(err)
	}
}

package com.algolia;

import com.algolia.api.SearchClient;
import com.algolia.config.*;
import com.algolia.model.search.*;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;

class saveImageClassifications {

  static class Image {

    public String imageURL;
    public String objectID;
    public List<Map<String, Object>> objects;

    public Image(String imageURL, String objectID, List<Map<String, Object>> objects) {
      this.imageURL = imageURL;
      this.objectID = objectID;
      this.objects = objects;
    }
  }

  // Retrieve labels
  static Image getImageLabels(String imageURL, String objectID, float scoreLimit) {
    // Implement your image classification logic here
    return new Image(imageURL, objectID, new ArrayList<>());
  }

  public static void main(String[] args) throws Exception {
    // API key ACL should include editSettings / addObject
    try (SearchClient client = new SearchClient("ALGOLIA_APPLICATION_ID", "ALGOLIA_API_KEY")) {
      var hits = client.browseObjects("INDEX_NAME", Image.class);

      List<Image> records = new ArrayList<>();
      for (var hit : hits) {
        var image = getImageLabels(hit.imageURL, hit.objectID, 0.5f);
        records.add(image);
      }

      // Update records with image classifications
      client.partialUpdateObjects("INDEX_NAME", records, true);
    } catch (Exception e) {
      System.out.println(e.getMessage());
    }
  }
}

import { algoliasearch } from 'algoliasearch';

interface Image extends Record<string, unknown> {
  imageURL: string;
  objectID: string;
  objects: Record<string, unknown>[];
}

async function saveImageClassifications() {
  // Retrieve labels
  function getImageLabels(_imageURL, _objectID, _scoreLimit): Image {
    // Implement your image classification logic here
    return { imageURL: '', objectID: '', objects: [] };
  }

  // API key ACL should include editSettings / addObject
  const client = algoliasearch('ALGOLIA_APPLICATION_ID', 'ALGOLIA_API_KEY');

  const records: Image[] = [];

  await client.browseObjects<Image>({
    indexName: 'INDEX_NAME',
    browseParams: undefined,
    aggregator: (browseResponse) => {
      records.push(
        ...browseResponse.hits.map((hit) => {
          return getImageLabels(hit.imageURL, hit.objectID, 0.5);
        }),
      );
    },
  });

  // Update records with image classifications
  await client.partialUpdateObjects({ indexName: 'products', objects: records, createIfNotExists: true });
}

import com.algolia.client.api.SearchClient
import com.algolia.client.configuration.*
import com.algolia.client.extensions.*
import com.algolia.client.model.search.BrowseParamsObject
import com.algolia.client.transport.*
import kotlinx.serialization.json.Json
import kotlinx.serialization.json.encodeToJsonElement
import kotlinx.serialization.json.jsonObject

suspend fun saveImageClassifications() {
  data class Image(val imageURL: String, val objectID: String, val objects: List<Map<String, Any>>)

  // Retrieve labels
  fun getImageLabels(imageURL: String, objectID: String, scoreLimit: Float): Image {
    // Implement your image classification logic here
    return Image(imageURL, objectID, emptyList())
  }

  // API key ACL should include editSettings / addObject
  try {
    val client = SearchClient(appId = "ALGOLIA_APPLICATION_ID", apiKey = "ALGOLIA_API_KEY")

    val images: MutableList<Image> = mutableListOf()

    client.browseObjects(
      indexName = "INDEX_NAME",
      params = BrowseParamsObject(),
      aggregator = { browseResponse ->
        images.addAll(
          browseResponse.hits.map {
            val props = it.additionalProperties ?: emptyMap()
            return@map getImageLabels(
              props.getOrElse("imageURL") { "" }.toString(),
              it.objectID,
              0.5f,
            )
          }
        )
      },
    )

    val records = images.map { Json.encodeToJsonElement(it).jsonObject }

    // Update records with image classifications
    client.partialUpdateObjects(
      indexName = "INDEX_NAME",
      objects = records,
      createIfNotExists = true,
    )
  } catch (e: Exception) {
    println(e.message)
  }
}

<?php

use Algolia\AlgoliaSearch\Api\SearchClient;

$getImageLabels = function ($imageURL, $objectID, $scoreLimit) {
    // Implement your image classification logic here
    return ['objectID' => '', 'imageURL' => '', 'objects' => []];
};

$client = SearchClient::create('ALGOLIA_APPLICATION_ID', 'ALGOLIA_API_KEY');

$records = [];

$hits = $client->browseObjects('INDEX_NAME');
foreach ($hits as $hit) {
    $imageURL = $hit['imageURL'];
    $records[] = $getImageLabels($imageURL, $hit['objectID'], 0.5);
}

$client->partialUpdateObjects(
    'INDEX_NAME',
    $records,
    true,
);

from algoliasearch.search.client import SearchClientSync

from algoliasearch.search.models.browse_response import BrowseResponse


def _get_image_labels(image_url, object_id, score_limit):
    # Implement your image classification logic here
    return {"objectID": "", "imageURL": "", "objects": []}


_client = SearchClientSync("ALGOLIA_APPLICATION_ID", "ALGOLIA_API_KEY")


records = []


def _aggregator(res: BrowseResponse):
    for hit in res.hits:
        props = hit.to_dict()
        image_url = props["imageURL"]
        records.append(_get_image_labels(image_url, hit.object_id, 0.5))


_client.browse_objects("INDEX_NAME", aggregator=_aggregator)

_client.partial_update_objects(
    index_name="INDEX_NAME",
    objects=records,
    create_if_not_exists=True,
)

require "algolia"

def get_image_labels(image_url, object_id, score_limit)
  # Implement your image classification logic here
  {"objectID" => "", "imageURL" => "", "objects" => []}
end

client = Algolia::SearchClient.create("ALGOLIA_APPLICATION_ID", "ALGOLIA_API_KEY")

records = []

client.browse_objects(
  "INDEX_NAME"
) do |resp|
  resp.hits.each do |hit|
    image_url = hit["imageURL"]
    records << get_image_labels(image_url, hit["objectID"], 0.5)
  end
end

client.partial_update_objects("INDEX_NAME", records, true)

import scala.concurrent.Future
import scala.concurrent.ExecutionContext.Implicits.global

import algoliasearch.api.SearchClient
import algoliasearch.config.*
import algoliasearch.extension.SearchClientExtensions
import algoliasearch.search.BrowseParamsObject

class Image(
    val imageURL: String,
    val objectID: String,
    val objects: Seq[Map[String, Any]]
)

def saveImageClassifications(): Future[Unit] = {
  // Retrieve labels
  def getImageLabels(imageURL: String, objectID: String, scoreLimit: Float): Image = {
    // Implement your image classification logic here
    Image(imageURL, objectID, Seq())
  }

  // API key ACL should include editSettings / addObject
  val client = SearchClient(appId = "ALGOLIA_APPLICATION_ID", apiKey = "ALGOLIA_API_KEY")

  val records: Seq[Image] = Seq.empty

  client.browseObjects(
    indexName = "INDEX_NAME",
    browseParams = BrowseParamsObject(),
    aggregator = { browseResponse =>
      records.appendedAll(
        browseResponse.hits.map { hit =>
          val props = hit.additionalProperties.getOrElse(List()).toMap
          getImageLabels(props.get("imageURL").toString, hit.objectID, 0.5f)
        }
      )
    }
  )

  // Update records with image classifications
  client
    .partialUpdateObjects(
      indexName = "INDEX_NAME",
      objects = records,
      createIfNotExists = true
    )
    .map { response =>
      println(response)
    }
    .recover { case ex: Exception =>
      println(s"An error occurred: ${ex.getMessage}")
    }
}

import Foundation
#if os(Linux) // For linux interop
    import FoundationNetworking
#endif

import AlgoliaCore
import AlgoliaSearch

func saveImageClassifications() async throws {
    struct Image: Codable {
        var objectID: String
        var imageURL: String
        var objects: [[String: AnyCodable]]
    }

    func getImageLabels(imageURL _: String, objectID _: String, scoreLimit _: Double) -> Image {
        // Implement your image classification logic here
        Image(objectID: "", imageURL: "", objects: [])
    }

    let client = try SearchClient(appID: "ALGOLIA_APPLICATION_ID", apiKey: "ALGOLIA_API_KEY")

    var records: [Image] = []

    try await client.browseObjects(
        indexName: "INDEX_NAME",
        browseParams: BrowseParamsObject(),
        aggregator: { (response: BrowseResponse<Image>) in
            records.append(contentsOf: response.hits.map { hit in
                let imageURL = hit.imageURL
                return getImageLabels(imageURL: imageURL, objectID: hit.objectID, scoreLimit: 0.5)
            })
        }
    )

    try await client.partialUpdateObjects(indexName: "INDEX_NAME", objects: records, createIfNotExists: true)
}

With this completed, each product record has a labels attribute:

JSON

{
  "title": "Men's Athletic Shirt",
  "objectID": "807281751",
  "imageURL": "https://images-na.ssl-images-amazon.com/images/I/41uIVaJOLdL.jpg",
  "labels":[
    {
      "description": "Outerwear",
      "score": 0.9513528
    }
  ]
}

To let users search by label descriptions as text, add labels.description to your searchableAttributes. To add search by image, or to filter on labels, include labels.description in attributesForFaceting.

Using ViSenze

This example uses the getImageLabels function from the classification section to retrieve labels for each record with browse. It then uses the partialUpdateObjects method to add the labels to the record. It then updates the index settings to include each object’s labels in attributesForFaceting and searchableAttributes.

namespace Algolia;

using System;
using System.Collections.Generic;
using System.Net.Http;
using System.Text.Json;
using Algolia.Search.Clients;
using Algolia.Search.Http;
using Algolia.Search.Models.Search;

class SaveImageClassificationsAndSettings
{
  class Image
  {
    public required string ImageUrl { get; set; }
    public required string ObjectId { get; set; }
    public required List<Dictionary<string, object>> Objects { get; set; }
  }

  // Retrieve labels
  async Task<Image> GetImageLabels(string imageURL, string objectID, float scoreLimit)
  {
    // Implement your image classification logic here
    return await Task.Run(() =>
      new Image
      {
        ImageUrl = "",
        ObjectId = "",
        Objects = [],
      }
    );
  }

  async Task Main(string[] args)
  {
    try
    {
      // API key ACL should include editSettings / addObject
      var client = new SearchClient(new SearchConfig("ALGOLIA_APPLICATION_ID", "ALGOLIA_API_KEY"));

      var hits = await client.BrowseObjectsAsync<Image>(
        "INDEX_NAME",
        new BrowseParamsObject()
      );

      var images = hits.ToList();
      var records = images
        .Select(hit => GetImageLabels(hit.ImageUrl, hit.ObjectId, 0.5f))
        .Select(src => src.Result)
        .ToList();

      // Update records with image classifications
      await client.PartialUpdateObjectsAsync("INDEX_NAME", records, true);

      List<string> facets = [];
      List<string> attributes = [];

      foreach (var image in images)
      {
        foreach (var obj in image.Objects)
        {
          foreach (var key in obj.Keys)
          {
            if (obj[key] is IEnumerable<object>)
            {
              facets.Add($"searchable(objects.{key}.label)");
              facets.Add($"searchable(objects.{key}.score)");
              attributes.Add($"objects.{key}.label");
            }
          }
        }
      }

      var currentSettings = await client.GetSettingsAsync("INDEX_NAME");

      var settings = new IndexSettings
      {
        SearchableAttributes = currentSettings.SearchableAttributes.Concat(attributes).ToList(),
        AttributesForFaceting = currentSettings.AttributesForFaceting.Concat(facets).ToList(),
      };

      await client.SetSettingsAsync("INDEX_NAME", settings);
    }
    catch (Exception e)
    {
      Console.WriteLine(e.Message);
    }
  }
}

import 'package:algolia_client_search/algolia_client_search.dart';

final class Image {
  final String imageUrl;
  final String objectID;
  List<Map<String, dynamic>> objects;

  Image(String imageUrl, String objectID, List<Map<String, dynamic>> objects)
      : imageUrl = imageUrl,
        objectID = objectID,
        objects = objects;
}

Future<Map<String, dynamic>> getImageLabels(
    String imageURL, String objectID, double scoreLimit) async {
  // Implement your image classification logic here
  return {};
}

void saveImageClassifications() async {
  // API key ACL should include editSettings / addObject
  final client =
      SearchClient(appId: 'ALGOLIA_APPLICATION_ID', apiKey: 'ALGOLIA_API_KEY');

  final List<Map<String, dynamic>> records = [];

  BrowseResponse? browseResponse;
  do {
    final browseParams =
        BrowseParamsObject(hitsPerPage: 1000, cursor: browseResponse?.cursor);
    browseResponse = await client.browse(
        indexName: "INDEX_NAME", browseParams: browseParams);
    for (final hit in browseResponse.hits) {
      final props = hit.toJson();
      final imageWithLabels =
          await getImageLabels(props["imageUrl"], hit.objectID, 0.5);
      records.add(imageWithLabels);
    }
  } while (browseResponse.cursor != null);

  final batchParams = BatchWriteParams(
      requests: records
          .map((record) => BatchRequest(
                action: Action.partialUpdateObject,
                body: record,
              ))
          .toList());

  // Update records with image classifications
  await client.batch(
    indexName: "INDEX_NAME",
    batchWriteParams: batchParams,
  );

  List<String> facets = [];
  List<String> attributes = [];

  for (var record in records) {
    record["objects"].forEach((object) {
      object.forEach((key, values) {
        if (values is Iterable) {
          facets.add("searchable(objects.$key.label)");
          facets.add("searchable(objects.$key.score)");
          attributes.add("objects.$key.label");
        }
      });
    });
  }

  final currentSettings = await client.getSettings(
    indexName: "INDEX_NAME",
  );

  final settings = IndexSettings(
    searchableAttributes:
        (currentSettings.searchableAttributes ?? []) + attributes,
    attributesForFaceting:
        (currentSettings.attributesForFaceting ?? []) + facets,
  );

  await client.setSettings(
    indexName: "INDEX_NAME",
    indexSettings: settings,
  );
}

package main

import (
	"fmt"

	"github.com/algolia/algoliasearch-client-go/v4/algolia/search"
)

func saveImageClassificationsAndSettings() {
	type Image struct {
		ImageURL string           `json:"imageURL"`
		ObjectID string           `json:"objectID"`
		Objects  []map[string]any `json:"objects"`
	}

	getImageLabels := func(imageURL, objectID string, scoreLimit float64) Image {
		// Implement your image classification logic here
		return Image{
			ImageURL: imageURL,
			ObjectID: objectID,
			Objects:  []map[string]any{},
		}
	}

	client, err := search.NewClient("ALGOLIA_APPLICATION_ID", "ALGOLIA_API_KEY")
	if err != nil {
		// The client can fail to initialize if you pass an invalid parameter.
		panic(err)
	}

	images := []Image{}

	err = client.BrowseObjects("INDEX_NAME", search.BrowseParamsObject{}, search.WithAggregator(func(res any, err error) {
		if err != nil {
			panic(err)
		}

		browseRes, ok := res.(search.BrowseResponse)
		if !ok {
			panic("Invalid response")
		}

		for _, hit := range browseRes.Hits {
			props := hit.AdditionalProperties
			imageURL, _ := props["imageURL"].(string)
			images = append(images, getImageLabels(imageURL, hit.GetObjectID(), 0.5))
		}
	}))
	if err != nil {
		panic(err)
	}

	records := make([]map[string]any, len(images))
	for i, img := range images {
		records[i] = map[string]any{
			"imageURL": img.ImageURL,
			"objectID": img.ObjectID,
			"objects":  img.Objects,
		}
	}

	_, err = client.PartialUpdateObjects(
		"INDEX_NAME", records, search.WithCreateIfNotExists(true))
	if err != nil {
		panic(err)
	}

	facets := []string{}
	attributes := []string{}

	for _, record := range images {
		for _, obj := range record.Objects {
			for key, values := range obj {
				if _, ok := values.([]any); ok {
					facets = append(facets,
						fmt.Sprintf("searchable(objects.%s.label)", key),
						fmt.Sprintf("searchable(objects.%s.score)", key),
					)
					attributes = append(attributes, fmt.Sprintf("objects.%s.label)", key))
				}
			}
		}
	}

	currentSettings, err := client.GetSettings(client.NewApiGetSettingsRequest(
		"INDEX_NAME"))
	if err != nil {
		panic(err)
	}

	settings := search.NewIndexSettings().
		SetSearchableAttributes(append(currentSettings.SearchableAttributes, attributes...)).
		SetAttributesForFaceting(append(currentSettings.AttributesForFaceting, facets...))

	_, err = client.SetSettings(client.NewApiSetSettingsRequest(
		"INDEX_NAME", settings))
	if err != nil {
		panic(err)
	}
}

package com.algolia;

import com.algolia.api.SearchClient;
import com.algolia.config.*;
import com.algolia.model.search.*;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;

class saveImageClassificationsAndSettings {

  static class Image {

    public String imageURL;
    public String objectID;
    public List<Map<String, Object>> objects;

    public Image(String imageURL, String objectID, List<Map<String, Object>> objects) {
      this.imageURL = imageURL;
      this.objectID = objectID;
      this.objects = objects;
    }
  }

  // Retrieve labels
  static Image getImageLabels(String imageURL, String objectID, float scoreLimit) {
    // Implement your image classification logic here
    return new Image(imageURL, objectID, new ArrayList<>());
  }

  public static void main(String[] args) throws Exception {
    // API key ACL should include editSettings / addObject
    try (SearchClient client = new SearchClient("ALGOLIA_APPLICATION_ID", "ALGOLIA_API_KEY")) {
      var hits = client.browseObjects("INDEX_NAME", Image.class);

      List<Image> records = new ArrayList<>();
      for (var hit : hits) {
        var image = getImageLabels(hit.imageURL, hit.objectID, 0.5f);
        records.add(image);
      }

      // Update records with image classifications
      client.partialUpdateObjects("INDEX_NAME", records, true);

      List<String> facets = new ArrayList<>();
      List<String> attributes = new ArrayList<>();

      records.forEach(record -> {
        record.objects.forEach(object -> {
          object.forEach((key, values) -> {
            if (values instanceof Iterable) {
              facets.add("searchable(objects." + key + ".label)");
              facets.add("searchable(objects." + key + ".score)");
              attributes.add("objects." + key + ".label");
            }
          });
        });
      });

      var currentSettings = client.getSettings("INDEX_NAME");

      var attributesForFaceting =
        currentSettings.getAttributesForFaceting() != null ? currentSettings.getAttributesForFaceting() : new ArrayList<String>();
      var searchableAttributes =
        currentSettings.getSearchableAttributes() != null ? currentSettings.getSearchableAttributes() : new ArrayList<String>();

      attributesForFaceting.addAll(facets);
      searchableAttributes.addAll(attributes);

      var settings = new IndexSettings().setAttributesForFaceting(attributesForFaceting).setSearchableAttributes(searchableAttributes);

      client.setSettings("INDEX_NAME", settings);
    } catch (Exception e) {
      System.out.println(e.getMessage());
    }
  }
}

import { algoliasearch } from 'algoliasearch';

interface Image extends Record<string, unknown> {
  imageURL: string;
  objectID: string;
  objects: Record<string, unknown>[];
}

async function saveImageClassifications() {
  // Retrieve labels
  function getImageLabels(_imageURL, _objectID, _scoreLimit): Image {
    // Implement your image classification logic here
    return { imageURL: '', objectID: '', objects: [] };
  }

  // API key ACL should include editSettings / addObject
  const client = algoliasearch('ALGOLIA_APPLICATION_ID', 'ALGOLIA_API_KEY');

  const records: Image[] = [];

  await client.browseObjects<Image>({
    indexName: 'INDEX_NAME',
    browseParams: undefined,
    aggregator: (browseResponse) => {
      records.push(
        ...browseResponse.hits.map((hit) => {
          return getImageLabels(hit.imageURL, hit.objectID, 0.5);
        }),
      );
    },
  });

  // Update records with image classifications
  await client.partialUpdateObjects({ indexName: 'products', objects: records, createIfNotExists: true });

  const facets: string[] = [];
  const attributes: string[] = [];

  records.forEach((record) => {
    record.objects.forEach((obj) => {
      Object.entries(obj).forEach(([key, values]) => {
        if (Array.isArray(values)) {
          // Checking if iterable
          facets.push(`searchable(objects.${key}.label)`);
          facets.push(`searchable(objects.${key}.score)`);
          attributes.push(`objects.${key}.label`);
        }
      });
    });
  });

  const currentSettings = await client.getSettings({ indexName: 'indexName' });

  const settings = {
    searchableAttributes: (currentSettings.searchableAttributes ?? []).concat(attributes ?? []),
    attributesForFaceting: (currentSettings.attributesForFaceting ?? []).concat(facets ?? []),
  };

  await client.setSettings({ indexName: 'indexName', indexSettings: settings });
}

import com.algolia.client.api.SearchClient
import com.algolia.client.configuration.*
import com.algolia.client.extensions.*
import com.algolia.client.model.search.BrowseParamsObject
import com.algolia.client.model.search.IndexSettings
import com.algolia.client.transport.*
import kotlinx.serialization.json.Json
import kotlinx.serialization.json.encodeToJsonElement
import kotlinx.serialization.json.jsonObject

suspend fun saveImageClassificationsAndSettings() {
  data class Image(val imageURL: String, val objectID: String, val objects: List<Map<String, Any>>)

  // Retrieve labels
  fun getImageLabels(imageURL: String, objectID: String, scoreLimit: Float): Image {
    // Implement your image classification logic here
    return Image(imageURL, objectID, emptyList())
  }

  // API key ACL should include editSettings / addObject
  try {
    val client = SearchClient(appId = "ALGOLIA_APPLICATION_ID", apiKey = "ALGOLIA_API_KEY")

    val images: MutableList<Image> = mutableListOf()

    client.browseObjects(
      indexName = "INDEX_NAME",
      params = BrowseParamsObject(),
      aggregator = { browseResponse ->
        images.addAll(
          browseResponse.hits.map {
            val props = it.additionalProperties ?: emptyMap()
            return@map getImageLabels(
              props.getOrElse("imageURL") { "" }.toString(),
              it.objectID,
              0.5f,
            )
          }
        )
      },
    )

    val records = images.map { Json.encodeToJsonElement(it).jsonObject }

    // Update records with image classifications
    client.partialUpdateObjects(
      indexName = "INDEX_NAME",
      objects = records,
      createIfNotExists = true,
    )

    val facets = mutableListOf<String>()
    val attributes = mutableListOf<String>()

    images.forEach { record ->
      record.objects.forEach { obj ->
        obj.forEach { (key, values) ->
          if (values is Iterable<*>) {
            facets.add("searchable(objects.$key.label)")
            facets.add("searchable(objects.$key.score)")
            attributes.add("objects.$key.label")
          }
        }
      }
    }

    val currentSettings = client.getSettings(indexName = "INDEX_NAME")

    val settings =
      IndexSettings(
        searchableAttributes = currentSettings.searchableAttributes?.plus(attributes),
        attributesForFaceting = currentSettings.attributesForFaceting?.plus(facets),
      )

    client.setSettings(indexName = "INDEX_NAME", indexSettings = settings)
  } catch (e: Exception) {
    println(e.message)
  }
}

<?php

use Algolia\AlgoliaSearch\Api\SearchClient;
use Algolia\AlgoliaSearch\Model\Search\IndexSettings;

$getImageLabels = function ($imageURL, $objectID, $scoreLimit) {
    // Implement your image classification logic here
    return ['objectID' => '', 'imageURL' => '', 'objects' => []];
};

$client = SearchClient::create('ALGOLIA_APPLICATION_ID', 'ALGOLIA_API_KEY');

$records = [];

$hits = $client->browseObjects('INDEX_NAME');
foreach ($hits as $hit) {
    $imageURL = $hit['imageURL'];
    $records[] = $getImageLabels($imageURL, $hit['objectID'], 0.5);
}

$client->partialUpdateObjects(
    'INDEX_NAME',
    $records,
    true,
);

$facets = [];
$attributes = [];

foreach ($records as $record) {
    foreach ($record['objects'] as $obj) {
        foreach ($obj as $key => $values) {
            if (is_array($values)) {
                $facets[] = "searchable(objects.{$key}.label)";
                $facets[] = "searchable(objects.{$key}.score)";
                $attributes[] = "objects.{$key}.label)";
            }
        }
    }
}

$currentSettings = $client->getSettings(
    'INDEX_NAME',
);

$settings = (new IndexSettings())
    ->setSearchableAttributes(array_merge($currentSettings->searchableAttributes ?? [], $attributes))
    ->setAttributesForFaceting(array_merge($currentSettings->attributesForFaceting ?? [], $facets))
;

$client->setSettings(
    'INDEX_NAME',
    $settings,
);

from algoliasearch.search.client import SearchClientSync

from algoliasearch.search.models.browse_response import BrowseResponse
from algoliasearch.search.models.index_settings import IndexSettings


def _get_image_labels(image_url, object_id, score_limit):
    # Implement your image classification logic here
    return {"objectID": "", "imageURL": "", "objects": []}


_client = SearchClientSync("ALGOLIA_APPLICATION_ID", "ALGOLIA_API_KEY")


records = []


def _aggregator(res: BrowseResponse):
    for hit in res.hits:
        props = hit.to_dict()
        image_url = props["imageURL"]
        records.append(_get_image_labels(image_url, hit.object_id, 0.5))


_client.browse_objects("INDEX_NAME", aggregator=_aggregator)

_client.partial_update_objects(
    index_name="INDEX_NAME",
    objects=records,
    create_if_not_exists=True,
)

facets = []
attributes = []

for record in records:
    for obj in record["objects"]:
        for key, values in obj.items():
            if isinstance(values, list):
                facets.extend(
                    [
                        f"searchable(objects.{key}.label)",
                        f"searchable(objects.{key}.score)",
                    ]
                )
                attributes.append(f"objects.{key}.label)")

current_settings = _client.get_settings(
    index_name="INDEX_NAME",
)

settings = IndexSettings(
    searchable_attributes=(current_settings.searchable_attributes or []) + attributes,
    attributes_for_faceting=(current_settings.attributes_for_faceting or []) + facets,
)

_client.set_settings(
    index_name="INDEX_NAME",
    index_settings=settings,
)

require "algolia"

def get_image_labels(image_url, object_id, score_limit)
  # Implement your image classification logic here
  {"objectID" => "", "imageURL" => "", "objects" => []}
end

client = Algolia::SearchClient.create("ALGOLIA_APPLICATION_ID", "ALGOLIA_API_KEY")

records = []

client.browse_objects(
  "INDEX_NAME"
) do |resp|
  resp.hits.each do |hit|
    image_url = hit["imageURL"]
    records << get_image_labels(image_url, hit["objectID"], 0.5)
  end
end

client.partial_update_objects("INDEX_NAME", records, true)

facets = []
attributes = []

records.each do |record|
  record["objects"].each do |obj|
    obj.each do |key, values|
      if values.is_a?(Array)
        facets.push(
          "searchable(objects.#{key}.label)",
          "searchable(objects.#{key}.score)"
        )
        attributes << "objects.#{key}.label)"
      end
    end
  end
end

current_settings = client.get_settings("INDEX_NAME")

settings = Algolia::Search::IndexSettings.new(
  searchable_attributes: (current_settings.searchable_attributes || []) + attributes,
  attributes_for_faceting: (current_settings.attributes_for_faceting || []) + facets
)

client.set_settings("INDEX_NAME", settings)

import scala.concurrent.{Await, Future}
import scala.concurrent.duration.Duration
import scala.concurrent.ExecutionContext.Implicits.global

import algoliasearch.api.SearchClient
import algoliasearch.config.*
import algoliasearch.extension.SearchClientExtensions
import algoliasearch.search.{BrowseParamsObject, IndexSettings}

def saveImageClassificationsAndSettings(): Future[Unit] = {
  // Retrieve labels
  def getImageLabels(imageURL: String, objectID: String, scoreLimit: Float): Image = {
    // Implement your image classification logic here
    Image(imageURL, objectID, Seq())
  }

  // API key ACL should include editSettings / addObject
  val client = SearchClient(appId = "ALGOLIA_APPLICATION_ID", apiKey = "ALGOLIA_API_KEY")

  var records: Seq[Image] = Seq.empty

  client.browseObjects(
    indexName = "INDEX_NAME",
    browseParams = BrowseParamsObject(),
    aggregator = { browseResponse =>
      records = records.appendedAll(
        browseResponse.hits.map { hit =>
          val props = hit.additionalProperties.getOrElse(List()).toMap
          getImageLabels(props.get("imageURL").toString, hit.objectID, 0.5f)
        }
      )
    }
  )

  // Update records with image classifications
  client.partialUpdateObjects(
    indexName = "INDEX_NAME",
    objects = records,
    createIfNotExists = true
  )

  var facets: Seq[String] = Seq.empty
  var attributes: Seq[String] = Seq.empty

  records.foreach(record => {
    record.objects.foreach(obj => {
      obj.foreach { case (key, value) =>
        if (value.isInstanceOf[scala.collection.Iterable[?]]) {
          facets = facets.appended(s"searchable(objects.${key}.label)")
          facets = facets.appended(s"searchable(objects.${key}.score)")
          attributes = attributes.appended(s"objects.${key}.label")
        }
      }
    })
  })

  val currentSettings = Await.result(
    client.getSettings(
      indexName = "INDEX_NAME"
    ),
    Duration(5, "sec")
  )

  val settings = IndexSettings(
    attributesForFaceting = Some(currentSettings.attributesForFaceting.getOrElse(Seq.empty) ++ facets),
    searchableAttributes = Some(currentSettings.searchableAttributes.getOrElse(Seq.empty) ++ attributes)
  )

  client
    .setSettings(
      indexName = "INDEX_NAME",
      indexSettings = settings
    )
    .map { response =>
      println(response)
    }
    .recover { case ex: Exception =>
      println(s"An error occurred: ${ex.getMessage}")
    }
}

import Foundation
#if os(Linux) // For linux interop
    import FoundationNetworking
#endif

import AlgoliaCore
import AlgoliaSearch

func saveImageClassificationsAndSettings() async throws {
    struct Image: Codable {
        var objectID: String
        var imageURL: String
        var objects: [[String: AnyCodable]]
    }

    func getImageLabels(imageURL _: String, objectID _: String, scoreLimit _: Double) -> Image {
        // Implement your image classification logic here
        Image(objectID: "", imageURL: "", objects: [])
    }

    let client = try SearchClient(appID: "ALGOLIA_APPLICATION_ID", apiKey: "ALGOLIA_API_KEY")

    var records: [Image] = []

    try await client.browseObjects(
        indexName: "INDEX_NAME",
        browseParams: BrowseParamsObject(),
        aggregator: { (response: BrowseResponse<Image>) in
            records.append(contentsOf: response.hits.map { hit in
                let imageURL = hit.imageURL
                return getImageLabels(imageURL: imageURL, objectID: hit.objectID, scoreLimit: 0.5)
            })
        }
    )

    try await client.partialUpdateObjects(indexName: "INDEX_NAME", objects: records, createIfNotExists: true)

    var facets: [String] = []
    var attributes: [String] = []

    for record in records {
        for obj in record.objects {
            for (key, values) in obj {
                if values.value is [Any] {
                    facets.append(contentsOf: [
                        "searchable(objects.\(key).label)",
                        "searchable(objects.\(key).score)",
                    ])
                    attributes.append("objects.\(key).label)")
                }
            }
        }
    }

    let currentSettings = try await client.getSettings(indexName: "INDEX_NAME")

    let settings = IndexSettings(
        attributesForFaceting: (currentSettings.attributesForFaceting ?? []) + facets,
        searchableAttributes: (currentSettings.searchableAttributes ?? []) + attributes
    )

    try await client.setSettings(indexName: "INDEX_NAME", indexSettings: settings)
}

With this completed, your records contain the recognized objects with their coordinates, labels, and scores.

JSON

{
  "imageURL": "https://images-na.ssl-images-amazon.com/images/I/41uIVaJOLdL.jpg",
  "objectID": "439784001",
  "objects": [
      {
        "x1": 65,
        "y1": 14,
        "x2": 729,
        "y2": 788,
        "apparel": [{ "label": "upper_body_garment", "score": 0.98 }],
        "product_color": [{ "label": "blue", "score": 0.91 }],
        "neckline": [{ "label": "round_neck", "score": 0.81 }]
      }
  ]
}

Next steps

Enriching your data with image classifications can improve search and discovery. You can take it further by letting users search for products with images.

Search and Discovery platform

Optimization and Personalization

AI-powered experiences

Production and scale

Image classification and tagging

Why use image classification?

What does image classification and tagging entail?

Platform considerations

Before you begin

Image classification

Using Google Vision API

Using ViSenze

Indexing image classifications

Using Google Vision API

Using ViSenze

Next steps

​Why use image classification?

​What does image classification and tagging entail?

​Platform considerations

​Before you begin

​Image classification

​Using Google Vision API

​Using ViSenze

​Indexing image classifications

​Using Google Vision API

​Using ViSenze

​Next steps

Why use image classification?

What does image classification and tagging entail?

Platform considerations

Before you begin

Image classification

Using Google Vision API

Using ViSenze

Indexing image classifications

Using Google Vision API

Using ViSenze

Next steps