The notion that the difficult part is linguistic is misguided; once both the query and document are represented in a common format, we would expect relevance to follow. But this isn't the case. Cross-Language Information Retrieval (CLIR) formally defines cross-lingual search as a retrieval task where the goal is to rank candidate documents in another language based on how relevant they are to the query. In their comprehensive survey of multilingual retrieval systems, Peters, Braschler & Clough define CLIR as an inherently retrieval task. Users formulate their information needs in one language and then have the system retrieve relevant documents in another. Although translation is a necessary component for establishing the linguistic bridge, it is not the entire system. Instead, translation provided the foundation for the linguistic bridge, while ranking determines what appears at position #1.

Solving the first problem does not necessarily equate to solving the second.

For years, researchers at CLIR have demonstrated that there has never been a bottleneck related to the quality of the translation itself. There has always been a bottleneck, though: retrieval precision after transformation. Translation is about developing a format that finds all the "good" documents (i.e., relevance) and not how well those good documents are translated into a human readable format. This is a retrieval issue. Systems that have optimized their translations and have not optimized their ranking will always do poorly compared to systems that have optimized their ranking. Surface form is no longer an issue; position is.

Multilingual search is therefore not optimal for the global e-commerce platforms, SaaS products, or content-heavy applications. What decision-makers really want are the cost of indexing, the cost of the language model, relevance parity across multiple languages/regions, operational complexity of managing configuration options for each locale, and speed-to-market once a company determines it wants to enter a new language ecosystem.

A platform that ranks appropriately in English, but incorrectly in French, will lose money in France. And this loss-of-money can be measured. But the correct ranking of the translation layer that sits atop it does not offset the incorrect ranking.

A key difference needs to exist between the NLP functions (language embedding and/or optional translations) and the functions we may refer to as “real time linguistic detection” and then the NLP sends the locale identifier, vector, and normalized query string to the retrieval layer. The retrieval layer will use those inputs to apply language aware tokenization, filtering, etc., and produce deterministic and explainable results at scale. Although Google's BERT implementation has enhanced the understanding of the query before it is passed to the retrieval layer, it still impacts the way the ranking occurs because the ranking mechanism remains unchanged and independent from the language understanding mechanism.

It is also important to maintain the sequential aspect of the language understanding and ranking layers, and combining them will negatively impact both layers.

In addition, there are not multiple relevance layers serving all the language markets, and therefore each language market must solve this problem independently. Each locale must be able to handle multiple indexes, ad hoc translation pipes, and multiple synonym lists. Additionally, the resultant ranking rules will be trained per market, but this does not ensure that the results are consistent among the various markets.

Finally, the increase in complexity for supporting multiple languages increases proportionally to the number of languages supported, without a particular design architecture for multi-lingual indexing as illustrated in Microsoft Azure AI Search Documentation. Therefore, a platform supporting ten languages, without a central governing body for relevance, is operating ten separate relevance problems simultaneously.

Algolia fit: the five signals unified

Algolia AI Search utilizes the output of the NLP as the layer that transforms the text; thus, it provides precise relevance in the context of multilingual relevance. Five types of heterogeneous signals are used to provide a unique type of information that cannot be produced using a single NLP primitive.

Algolia provides a single platform for management of the five signals affecting the user search experience across numerous languages; it will utilize the user’s native language as the basis for generating the morphological rules for tokenizing the user’s input; and it will transmit queries to configurations that match the user’s locale (fr-CA and fr-FR have different synonym lists, promotional rule sets, and accessible filters), so the correct configuration is returned for each locale. Algolia will adjust the ranking factor weights in the algorithm, based on the semantic meaning of the query to help determine what the user wants to accomplish with their search – browse, purchase, or research. Algolia also accounts for regional cultural differences within the individual regions it operates in through product naming conventions, seasonal marketing windows, and how branding and loyalty programs are perceived in those regions, none of which would be present in other regions. Additionally, by segmenting click and conversion data from aggregate global data segmented by locale, there is no opportunity for the global traffic rich market segments to influence search results outside of the locales where they generate traffic. The logic of creating a single rank computation is uniformly applied to create search results for each market.

Technical implementation: relevance parity signal schema

The data model includes all necessary information to generate an indexed multilingual product record containing locale signal and language metadata to enable equally weighted rankings across multiple markets. Since language-aware processing is configured at the index level (via settings like indexLanguages and ignorePlurals), tokenizing, stemming, and stop word elimination reflect the linguistic attributes of each language without requiring per-field analyzer assignments in application code. All of the business logic fields (margin weight, availability, and promotional priority) are language neutral, and as such, they will generate the same outcome when queried by locale, and thus establish uniform business logic to support the ranking of search results across all markets.

/**
 * Multilingual Product Record Schema
 * Indexed in AI Search to support relevance parity across language markets.
 * Language-specific fields utilize settings applied via index-level configuration.
 * Business logic fields are locale-agnostic and applied consistently across all markets.
 */
{
  "objectID": "product_carry_on_luggage_001",
  "locale_signals": {
    "primary_language": "en",
    "supported_locales": ["en-US", "fr-FR", "ja-JP"],
    "bcp47_tag": "en-US"
  },
  "content": {
    "title_en": "Carry-on luggage, 22-inch hardshell",
    "title_fr": "Bagage cabine, coque rigide 22 pouces",
    "title_ja": "機内持ち込みスーツケース、22インチハードシェル",
    "description_en": "Lightweight hardshell carry-on approved for major airlines.",
    "description_fr": "Bagage cabine léger approuvé par les principales compagnies aériennes.",
    "description_ja": "主要航空会社に対応した軽量ハードシェルの機内持ち込み荷物。"
  },
  "ranking_signals": {
    "margin_weight": 0.82,
    "availability_score": 1.0,
    "promotional_priority": 2,
    "market_popularity": {
      "en-US": 9800,
      "fr-FR": 4200,
      "ja-JP": 3100
    }
  },
  // Illustrative: in production, analyzer assignments live in index settings, not the record
  "language_analyzer_config": {"en": "english", "fr": "french", "ja": "japanese" }
}

At the time the search takes place the routing layer uses the BCP 47 tag that it has determined was resolved for the user's session (via browser locale, account preference or language detection via upstream service), to route the search query to the appropriate Language Column. All other business logic columns will be applied unchanged. For example, users in all three markets will receive results with the same margin weights, availability scores & promotional priorities; which were normalized using the correct morphology for each of their respective languages. Therefore relevance parity is enforced at the schema level (i.e., database level); rather than assembled at runtime from multiple independent localization pipelines.

2. NeuralSearch: the hybrid engine for global precision

Optimizing for semantic similarity between vectors and search results within an embedding space does not always equate to identifying the most relevant search result for a given user with a particular business objective, in a specific geographic location. A user searching for “waterproof running shoes” in Japanese would receive their top ranked results based upon how close their query term is to theirs in the embedding space (trail shoes, water resistant shoes, outdoor footwear) as opposed to what products are available to be purchased, which have the largest profit margin, or are currently being promoted. Keyword matches provide precise match results whereas embeddings provide semantically related results; however, for a majority of commercial searches (e.g. product names, model numbers, category terms that must appear exactly), this trade off is wrong.

Google's BERT demonstrates how NLP can be used inside search engines' frameworks. For example, BERT knew that when you ask "Do estheticians stand a lot at work?," that “stand " was referring to the physical demands of the job, not "stand alone," which a keyword-based retrieval system would never know. Google didn't get rid of their retrieval architecture for BERT; they used language understanding as a feature of the retrieval process. They still have the ranker that they had before they used BERT and it functions in the exact same way. That's the model. NLP allows for many other features (i.e., translation, generating embeddings, determining the language of the query, etc.) The retrieval layer also allows for other processes such as indexing, ranking, filtering, and business logic. If you collapse the two layers into just a measure of how similar the vector representations are, then you'll probably create a system that will understand the query much better but will probably produce less relevant answers.

The way it has been implemented in practice is shown below. First, a user types a query in Spanish and the NLP identifies the language of the input by utilizing fastText’s lid.176 (with 176 languages based on a 917kb binary) and optionally creates cross-language embeddings via LASER or multi-language sentence transformers. This output is then passed downstream. Next, Algolia performs a search on multiple indexes that are multilingual, and uses rules for locale aware ranking, and applies business logic (i.e., availability, weight of margins, priority of promotions, etc.,) to determine the order of the returned items in the appropriate language. NLP provided the language understanding component. Algolia provided the final result. NeuralSearch provides both the semantic and keyword modes of operation concurrently, where they are fused together at the ranking level via a single API call. Neither channel will degrade the performance of the other. Adding cross-language semantic capabilities to a multilingual index currently being used does not detract from the ability to provide exact term matching that is required for commercial queries.

Cross-lingual embeddings train models across typologically distant languages (e.g. Japanese & English, Arabic & English, Chinese & English) and not just European languages that have a high degree of vocabulary overlap to allow for faster searching; they also train models on language pairs like English & Chinese where the writing systems, morphology, and syntax differ greatly from one another. Using a single encoder in LASER, which was trained on 223 million parallel sentences paired with either English or Spanish, a shared BPE dictionary is used among all languages. Therefore, lower-resource languages may utilize the training data of higher-resource languages. NeuralSearch's semantic channel works throughout this range of languages. When a user searches for "chaussures de course imperméables" in French, they receive the same search results as when a user searches for "waterproof running shoes" in English and the search results will be ordered based on the same logic used by the business. Meaning, it’s not the distance of the embedded object to an arbitrary reference point in a vector space that determines this ordering.

Algolia's proprietary NeuralSearch tech combines vector-based natural language processing and keyword matching in a single API. Most vector search engines, including those operating at enterprise scales like Algolia, use hybrid approaches to perform searches through keyword-based searches and/or natural language processing (nlp) that allow for the precision-recall trade off experienced by all vector search engines. Algolia can create completely separate indexes for operational/management purposes; in addition, since latency is a measurable revenue constraint in Algolia's business model and not an engineering preference, Algolia will run both retrieval modes in parallel without additional cost.

The platform will apply three constraints on AI search as a relevance platform globally that may never be enforced through nlp: consistent application of ranking logic across different markets, explainability of results through deterministic auditable rules, and scalability to the enterprise level. Translation and language models are used upstream to do their best work; downstream, Algolia puts into operation the results from them. It does not substitute translation and language models; instead it ensures that the results produced by them produce the desired outcomes.

Technical implementation: hybrid retrieval configuration for multilingual indexes

The Hybrid Configuration is utilized to enable neural search functionality to a multilingual Product Index via the use of both semantic vector-based retrieval and keyword-based search in the context of language (i.e., language-aware). This configuration enables the vector channel via semanticSearch while searchableAttributes list keyword channel as language-specific fields are ordered consistent with the locale of the user's session. The customRanking array allows the application of consistent business logic to all results returned by either retrieval method; thus ensuring that commercial intent will drive the ultimate ranking of a given result regardless of the retrieval channel employed to surface that result.

/**
 * NeuralSearch Hybrid Configuration for Multilingual Index
 * Combines semantic vector retrieval with language-aware keyword matching.
 * Business logic signals apply consistently across both retrieval channels.
 * Locale-specific fields are prioritized by query routing at search time.
 */
const indexSettings = {
  // Activate hybrid retrieval: semantic + keyword in one relevance stack
  semanticSearch: {
    eventSources: ["click", "conversion"],
  },

  // Language-specific fields ordered by retrieval priority
  // Active field set overridden at query time by locale-aware routing function
  searchableAttributes: [
    "title_en, title_fr, title_ja",       // Unordered: equivalent priority
    "description_en, description_fr, description_ja",
    "category_tags"                       // Language-agnostic taxonomy
  ],

  // Business logic applied consistently across all locale-aware queries
  customRanking: [
    "desc(promotional_priority)",
    "desc(availability_score)",
    "desc(margin_weight)",
    "desc(market_popularity.value)"       // Locale-specific popularity injected at query time
  ],

  // Typo tolerance configured per language morphology
  // Japanese and CJK scripts require typo tolerance disabled: no whitespace delimiters
  typoTolerance: true,
  disableTypoToleranceOnAttributes: ["title_ja", "description_ja"],  // CJK: no whitespace delimiters
  minWordSizefor1Typo: 5,
  minWordSizefor2Typos: 9,

  // Distinct product grouping across locale variants
  distinct: true,
  attributeForDistinct: "product_base_id"
};

/**
 * Locale-Aware Query Execution
 * Injects BCP 47 tag and market popularity signal at search time.
 * Routes semantic and keyword retrieval to locale-appropriate fields.
 */
async function searchMultilingual(query, userLocale) {
  const localeFieldMap = {
    "en-US": { titleField: "title_en", descField: "description_en" },
    "fr-FR": { titleField: "title_fr", descField: "description_fr" },
    "ja-JP": { titleField: "title_ja", descField: "description_ja" }
  };

  const { titleField, descField } = localeFieldMap[userLocale] || localeFieldMap["en-US"];

  return await index.search(query, {
    restrictSearchableAttributes: [titleField, descField, "category_tags"],
    // Popularity ranking applied via customRanking in index settings (locale-agnostic),
    ruleContexts: [userLocale],   // Activates locale-specific Query Rules
    analytics: true,
    clickAnalytics: true
  });
}

What allows one Index to be able to functionally represent multiple indexes based on the locale that the query originates from is the ruleContexts parameter. So, when the Japanese language query enters Algolia with the 'ja-JP' tag, only the QueryRules that are configured for that specific context will be activated (e.g., localized category boosts, locale-specific synonyms expansions, etc.), and all the other QueryRules will be inactivated. Likewise, the French user gets the French QueryRules and the American user gets the American QueryRules -- all from the same Index and processed through the same query processing pipeline. Thus, through the use of the ruleContexts parameter, the same Index can have varying behaviors per market -- without having to create additional infrastructure for each.

3. Linguistic boundaries: script variance and BCP 47 intent

Each Lucene language analyzer has its own way of handling the linguistic characteristics of its native language. Elasticsearch, providing access to Lucene's analyzer model at the field mapping level, provides an analyzer for each language based on different filter chains; it does not have a single filter chain and therefore a single set of filters that it switches or modifies when switching from one language to another. The French analyser is the most obvious representation of this idea; it utilizes the French-specific elision filter first to remove the contracted articles "l'", "m'", "t'", "d'", "j'", and "s'"; as such, if it did not do so prior to applying the stemmer, the tokens stemming would produce the same tokens as those that can be located in the uninflected index forms of words without articles. The French analyser filter chain is specified: french_elision → french_lowercase → french_stop → french_stemmer. Therefore, there are four sequential processes, each of which is necessary, and none of which could be replaced by the corresponding process in filter chains of other languages.

"rebuilt_french": {
  "tokenizer": "standard",
  "filter": [
    "french_elision",
    "lowercase",
    "french_stop",
    "french_stemmer"
  ]
}

In general, patterns observed with languages are similar, yet morphological issues vary dramatically from language to language. For example, the Arabic analyzer can normalize variant letter forms (e.g., multiple forms of alef, taa marbuta, and yaa) prior to stemming, since it has been observed that the same words represented in different orthographic forms produce false non-matches against an identical search query. The German analyzer, on the other hand, provides some limited inflectional light stemming, but does not apply decomposition to compound nouns as a default – i.e., "Handgepäckkoffer" will not be matched to a query for "Koffer," unless an optional decomposition filter is applied using external dictionary resources. Likewise, both Finnish and Turkish add grammatical information to stems, in ways that no English stemmer would ever solve.

Engineering leadership takeaway: All of the languages mentioned above have specific analyzers available within Elasticsearch and Lucene. In Lucene-based architectures, the critical task here is to assign each language-specific analyzer deliberately at the field level. [Note: this applies to Elasticsearch and Lucene deployments. In Algolia, language behavior is configured at the index level via indexLangauges and queryLanguages; no per-field analyzer assignment is required. As mentioned previously, the default analyzers fail silently across all of the above-mentioned languages – there are no errors or warnings; only incorrect results.

BCP 47 – The framework for constructing tags (defined in RFC 5646) and specifying matching rules (RFC 4647), provides locale tags with machine-readable locales and allows developers to create locale tags with language, script, region and variant subtags. According to the W3C guidelines, "you should keep the tag as short as you need to, adding subtags only if additional specificity is needed." For example, the French language spoken in Canada (fr-ca) differs significantly in vocabulary and spelling compared to France (fr-fr); and, the Chinese language traditionally spoken in Hong Kong (zh-hant-hk) differs significantly in the naming conventions for products, promotional calendars, and branding awareness compared to Taiwan (zh-hant-tw).

Search Engine Translation translates locale down to the level of granularity supported by your ranking logic and performs no additional translations. If your index cannot differentiate between en-us and en-gb at the ranker layer, then your index will provide no value in storing the region subtag. However, if your index is using different lists of synonyms, assigning different weights on promotion, or applying different filters on availability for users located in the United States vs. users in the UK, then the region subtag becomes a primary routing signal.

Due to the extreme differences in vocabulary, characters, and cultural reference points, the W3C BCP 47 standards defined the two tags (zh-hans, Simplified Chinese and zh-hant, Traditional Chinese) even though the languages share the same writing system. Each language is utilized very differently than the other, particularly in regards to the way they are written and the meaning of their respective words. That is why the W3C BCP 47 standards defined the two tags (zh-hans, Simplified Chinese and zh-hant, Traditional Chinese). Unfortunately, this difference can't be expressed through the language subtag alone.

Searches in the traditional Chinese language in Hong Kong may generate false negatives due to the lack of support for character variations; false positives generated by words with different meanings; and, popularity signals which are incorrectly calibrated to the wrong geographic area when searches are made against an index created for the Simplified Chinese language.

In some parts of Azerbaijan, Azerbaijani may be written in both Arabic and Latin scripts. In the development of an indexing strategy these alternatives would be taken into consideration and therefore in developing an indexing strategy there are alternatives to be considered when determining the correct approach to process the data before processing the data instead of correcting errors after the search query has been processed.

FastText's Lid .176 – 176 languages are extracted from a compressed binary model (Wikipedia, Tatoeba, and SETimes) to find upstream language and thus identify language in BCP 47 tags that go into the routing layer of Algolia's index engine when the session locale is unavailable to the search engine by using the raw query text, and the extraction of language and locale from the NLP layer and then moving it into the search retrieval layer is both an operational piece of the infrastructure and does not serve as metadata for display.

Algolia fit: linguistic boundaries

Algolia views locale as a key element in its Locale View Model, and therefore it does not view locale as secondary metadata on top of an Artificial Intelligence Search Model.

Algolia views the BCP 47 tag as the primary routing signal for routing the query via a rule in the Contexts parameter, which allows locale specific settings such as the expansion of synonyms, the weights assigned to the ranking logic and the business logic to segment marketing campaigns based on locale and cultural promotional rules for different markets.

Language-aware tokenization in Algolia is applied through index-level configuration (i.e., not per-field analyzer assignments) so morphological correctness is a design feature of the index rather than something negotiated in application code at runtime.

Algolia enables the integration of the five elements upon which multilingual relevance depends -- namely; the language (morphological tokenization) of the search terms, locale (to direct the query to the correct configuration for the marketplace), user intent (the ranking logic can be configured based on the semantic of the query), cultural expectations (promotional rules can be configured based on regional calendars and brand signals), and market-specific popularity (click and conversion signals are segmented by locale and not aggregated across all locales).

Technical implementation: relevance parity signal schema

The following configuration demonstrates how to apply language-aware settings to multilingual fields via index configuration (indexLanguages, queryLanguages) and BCP 47-scoped Query Rules for locale-aware synonym expansion. The Query Rules configuration demonstrates how a synonym expansion appropriate for British English is scoped to the en-GB locale context without affecting any other market.

/**
 * Language-Aware Index Field Configuration
 * Demonstrates language-aware processing configured at the index level. .
 * Ensures correct tokenization, stemming, and normalization per language.
 * Aligned with Lucene language analyzer architecture and Microsoft Azure AI Search
 * blended-index pattern for multilingual field design.
 */
const multilingualIndexSettings = {
  searchableAttributes: [
    "title_en",
    "title_fr",
    "title_ja",
    "title_de",
    "description_en",
    "description_fr",
    "description_ja",
    "description_de",
    "category_tags"   // Language-agnostic taxonomy, no language analyzer required
  ],
  // Index-level language configuration
  // Each language field processes queries through morphologically correct rules
  indexLanguages: ["en", "fr", "ja", "de"],
  queryLanguages: ["en", "fr", "ja", "de"],

  customRanking: [
    "desc(promotional_priority)",
    "desc(availability_score)",
    "desc(margin_weight)"
  ],

  // Ignore plurals and apply language-aware stemming per field
  ignorePlurals: ["en", "fr", "de"],  // Japanese morphology handled by segmentation

  // Remove stop words per language
  removeStopWords: ["en", "fr", "de"], // CJK languages do not use stop-word lists

  // Disable typo tolerance for CJK: character substitution carries semantic weight
  typoTolerance: true,
  disableTypoToleranceOnAttributes: ["title_ja"]
};

/**
 * BCP 47-Scoped Query Rule: locale-aware synonym expansion
 * Applies British English synonym mapping only when query context is en-GB.
 * Does not affect en-US, fr-FR, or any other locale context.
 * Configured via Algolia Query Rules API.
 */
const enGBSynonymRule = {
  objectID: "synonym-rule-en-gb-footwear",
  conditions: [  {
      context: "en-GB"   // BCP 47 region subtag as Query Rule scope
    }
  ],
  consequence: {
    filterPromotes: [
      {
        objectID: "category_footwear_uk",
        position: 0
      }
    ]
  }
};

/**
 * Language detection integration: fastText lid.176
 * Resolves BCP 47 tag from raw query text when session locale is unavailable.
 * Maps ISO 639 language code to BCP 47 tag for Query Rule context injection.
 */
async function resolveLocale(queryText, sessionLocale) {
  if (sessionLocale) return sessionLocale;
  // fastText lid.176 returns ISO 639 codes (e.g., "fr", "ja", "de")
  // Map to BCP 47 region-qualified tag for market-level routing
  const detectedLang = await fastTextModel.identify(queryText);
  const langToLocaleMap = {
    "en": "en-US",
    "fr": "fr-FR",
    "ja": "ja-JP",
    "de": "de-DE",
    "zh": "zh-Hans"   // Defaults to Simplified; override with script detection if required
  };

  return langToLocaleMap[detectedLang] || "en-US"; }

4. Indexing design: code-first architecture

Index design is the architectural decision that determines everything downstream.

• Pattern One: Segmented Indexes - Each Market has Its Own Index. Each index can be configured independently (for example analyzer settings, synonym lists, ranking rules, merchandising logic). There is no cross-language overlap at the time of query processing. The major advantage of segmented indexing is precision; there is no cross-language overlap in tokenization, no collision of synonyms across markets, and no adjustment of weights calibrated to rank products in one market which would affect how those products were ranked in another. The cost of replication will vary depending on the number of segments. In addition, with the creation of multiple indexes for each language, any changes made to the relevance model will need to be applied to every other index created for every other language. While this could still be managed with four languages, if you have fourteen languages, you will likely experience a configuration drift that will lead to a disintegration of the relevance parity which is the ultimate goal of the segmented indexing model. Right Choice when your markets require different ranking logic (different product catalogs, different pricing structures, different regulatory requirements).
• Pattern Two: Blended Indexes - One index, but fields are specific to a language. Product records will contain fields such as title_en, title_fr, title_ja, title_de as separate fields to search through and will utilize language-aware processing configured at the index level.. When an application searches for something, the routing layer retrieves the users locale tag from BCP 47 and then sends the search request to the field(s) for that locale. As described by Microsoft Azure AI Search: "Use the ‘searchFields’ parameter to send the search request to the field containing strings in the correct language and the ‘select’ parameter to only return results from the locale-specific fields, excluding fields in other languages. Operational efficiency of blended indexing is high, because you only need one index, one relevance configuration, and one telemetry surface. Right choice when cross-linguistic discovery is a feature of your products or when operational efficiency across many markets is a priority.

In addition to a catalog with five languages; Each language has ten searchable attributes. Therefore, prior to adding any non-language dependent attribute (price, availability, category taxonomies) to the catalog; there are fifty searchable fields in the catalog. At this degree of complexity, the schema is difficult to analyze and the routing logic for each field becomes extremely difficult to support when searching during a query. For example, if you were to enter a Japanese-language query; it would search all fifty fields, rather than just the five Japanese-language fields. This issue can be solved with a locale-to-field mapping table that is managed as an application configuration. The locale-to-field mapping table provides a single point of control for the routing behavior, is auditable through-out the entire code base, and does not require changes to the logic of the query.

The two architectural possibilities above will depend on whether both the documents and/or queries must be in multiple languages (Blended), or whether each market can be a separate product experience, using its own cataloging methodology and there would be no need for cross-lingual discovery (Segmented). The engineer’s manager is not committing an error in deciding which architecture to use; he/she is simply committing an error by default – i.e. he/she will choose the segmented approach first because it will appear less complicated than the blended approach and he/she will not account for the costs associated with indexing and replicating workflows across all ten markets, or he/she will choose the blended approach and will have no plan for how to name the fields, route the data, etc.

Engineering leadership has four areas of concern regarding engineering leadership: Index Cost/Size, Relevance Equivalence Across Markets, Complexity of Operations, and Time-to-Market for New Languages. These concerns are driven by the Index Design. To design an architecture of segmented indexes across ten different language markets would result in ten index configurations, ten synonym workflows, ten relevance tuning cycles and ten deployment pipelines. A single blended index can be used to deliver one of each of the items listed above. As you add additional language markets to your platform, the operational advantage will grow. The time to deploy a new language market will drop from months to days. Companies that run on top of seasonal commercial cycles (i.e., Golden Week, Singles' Day) will experience a direct positive impact to revenue.

Algolia fit: indexing design

Algolia’s AI Search supports both architectures by providing all possible configuration primitives to support both architectures. The configuration primitives (ignorePlurals, queryLanguages, indexLanguages, removeStopWords) can be configured at a segment level so that proper morphological analysis occurs for each query entered into the search box for each query. Furthermore, there is no additional overhead to provide the same configuration primitives for every query.

To support the creation of a locale-specific merchandising layer in blended deployments, Algolia employs index-level language configuration and search field routing based on rule-based context at query-time.

Therefore, when users use an index from Algolia, they will have a market-specific relevance surface that is derived from a single index. This results in there being only one model for all markets and therefore only one relevance model. Additionally, since there is not different ranking logic for each market, there is not one index per market, nor is there ad-hoc piping of translations to multiple indexes. Lastly, this eliminates irreconcilable relevance between regions.

A platform for global experiences that enforces consistency, explainability, and performance – not as properties the architecture tries to maintain, but as properties it produces.

Segmented index and blended index implementations

This segment provides details on the implementation of both of these configurations. The first is an example of implementing a segmented index configuration that can be used with a single language marketplace. All of the settings in this example are for the french morphology. The second is an example of a blended index query implemented using locale-aware field routing which follows the pattern of Microsoft Azure AI Search and the BCP 47 locale resolution established in section 3.

/**
 * Segmented Index Configuration: French Market
 * All settings tuned for French morphological rules.
 * Maintains full locale specificity at the cost of per-index operational overhead.
 * Recommended when French market requires distinct ranking logic,
 * catalog structure, or merchandising rules from other language markets.
 */
const frenchMarketIndexSettings = {
  // Language settings: French morphology throughout
  indexLanguages: ["fr"],
  queryLanguages: ["fr"],
  ignorePlurals: ["fr"],
  removeStopWords: ["fr"],

  searchableAttributes: [
    "title_fr",
    "description_fr",
    "category_tags",
    "brand_name"        // Brand names are language-agnostic
  ],

  customRanking: [
    "desc(promotional_priority)",
    "desc(availability_fr)",     // France-specific warehouse availability
    "desc(margin_weight_eur)",   // EUR-denominated margin signal
    "desc(popularity_fr)"        // French market popularity score
  ],

  // French-specific synonym configuration
  // Scoped entirely to this index: no cross-locale contamination
  synonyms: [
    {
      objectID: "syn-fr-baggage",
      type: "synonym",
      synonyms: ["bagage cabine", "bagage à main", "valise cabine"]
    },
    {
      objectID: "syn-fr-livraison",
      type: "oneWaySynonym",
      input: "livraison gratuite",
      synonyms: ["expédition offerte", "frais de port offerts"]
    }
  ],

  typoTolerance: true,
  minWordSizefor1Typo: 5,
  minWordSizefor2Typos: 9
};

/**
 * Blended Index Query Implementation
 * Single index serves multiple language markets via locale-aware field routing.
 * Implements Microsoft Azure AI Search searchFields routing pattern.
 * BCP 47 locale tag drives field selection, ruleContexts, and filter injection.
 * Recommended when cross-lingual discovery is a product requirement
 * or when operational efficiency across many language markets is the priority.
 */

// Locale-to-field routing table: single point of control for field selection
// Extend this map to add new language markets without modifying query logic
const localeFieldConfig = {
  "en-US": {
    searchFields:  ["title_en", "description_en", "category_tags"],
    selectFields:  ["title_en", "description_en", "price_usd", "availability"],
    popularityAttr: "popularity_en_us"
  },
  "fr-FR": {
    searchFields:  ["title_fr", "description_fr", "category_tags"],
    selectFields:  ["title_fr", "description_fr", "price_eur", "availability"],
    popularityAttr: "popularity_fr_fr"
  },
  "ja-JP": {
    searchFields:  ["title_ja", "description_ja", "category_tags"],
    selectFields:  ["title_ja", "description_ja", "price_jpy", "availability"],
    popularityAttr: "popularity_ja_jp"
  },
  "de-DE": {
    searchFields:  ["title_de", "description_de", "category_tags"],
    selectFields:  ["title_de", "description_de", "price_eur", "availability"],
    popularityAttr: "popularity_de_de"
  }
};

async function searchBlendedIndex(query, userLocale) {
  const config = localeFieldConfig[userLocale] || localeFieldConfig["en-US"];

  return await index.search(query, {
    // Route query to locale-appropriate language fields only
    restrictSearchableAttributes: config.searchFields,

    // Return only locale-appropriate fields in response payload
    attributesToRetrieve: config.selectFields,

    // Inject locale-specific popularity as optional ranking signal
// Popularity ranking applied via customRanking in index settings (locale-agnostic),

    // Activate locale-scoped Query Rules: synonyms, boosts, filters
    // French rules inactive for ja-JP context; Japanese rules inactive for fr-FR
    ruleContexts: [userLocale],

    // Pass locale to analytics for per-market relevance measurement
    analyticsTags: [`locale:${userLocale}`],
    analytics: true,
    clickAnalytics: true
  });
}

AnalyticsTags are used in BlendedQueries to quantify relevance parity instead of relying on the assumption of relevance parity by instrumenting every query's resolved locale as a tag to the analytics layer. By generating per-market clickthroughs, conversion rates and no result rates, these metrics can be directly compared to one another across markets. In the above example, if the ja-JP no result rate is 300% higher than the en-US no result rate, this would be immediately apparent in the telemetry data and could be addressed at the index configuration level (for example, missing synonyms or incorrect field routing) as well as identifying the current query pattern as unique to Japan and the current tokenization configuration does not properly process this query type. Relevance parity is an ongoing metric that needs to be constantly measured; it cannot be relied upon simply due to the presence of a good initial configuration.

5. The TCO advantage: interactive latency as an engineering constraint

Speed is a convertible constraint with financial implications: shaving .1 seconds from the user's search time adds 10 percent more money. Therefore, decisions that add delay to the search path are economically driven, not engineer-preference driven. The speed constraint is significantly tighter in multilingual searches as compared to monolingual searches because a naïve multi-lingual search process adds multiple delays to the search process: (a) there is a language-detection call prior to processing of the query; (b) there is an embedded generation of semantics for retrieval; (c) if the document was written in one language and the query is in a different language, then there may need to be a translation step; and (d) there is a ranking computation that includes locale-specific business rules on top of the basic relevance computation. These can all combine to produce what would be considered a "slow" search experience (versus an "interactive" one) and the lost revenue cannot be recouped simply because the correct language was returned.

Teams looking at build-versus-centralize decisions are considering all the engineering expenses directly associated with building a central vs. building multiple indexes, but they are also failing to properly consider and quantify the ongoing operational expenses (costs) of managing them. A self-managed multilingual architecture for use in eight different language markets means that there would have to be eight different configurations of indexes to synchronize, eight synonym lists to create and maintain, eight different rule sets to adjust as catalog data and marketplace conditions continue to evolve and eight separate deployment processes to oversee because any changes made to the centralized relevance model would need to be tested on each of the eight languages prior to being promoted into production. In this case the original team that built the application completed the project in six months but spent the next twenty-four months working on maintenance of the application; furthermore, the amount of money expended on maintenance increases by an additional amount with each new language market added to the business. The growth in maintenance expense can be further exacerbated due to index sprawl – when a product catalog consisting of ten million products is duplicated across eight indexes for different languages it will consume 8X the storage requirements of the same catalog represented by a single blended index; however, even if the blended index is configured appropriately there will be no performance gain regarding the ability to retrieve data from the index.

In terms of the development process, using an internal translation engine as part of a custom solution increases the costs associated with owning/operating the engine. Both language identification and cross-lingual embeddings can be achieved through the use of low-cost, low-maintenance NLP resources. fastText's lid.176 is a very small model that uses only 917 kB compressed to identify 176 languages. Multilingual models such as sentence-transformers provide cross-lingual embeddings and do not require a custom solution to generate them. Neither model requires a custom translation pipeline.

A managed relevance platform that provides native support for language aware tokenization will eliminate the need for a translation pipeline for both the tokenization and morphology layers. This reduces the number of NLP dependencies needed to produce a multilingual search experience to just two. The interval between making a business decision to offer a search experience in a new language market and having a production quality search experience available for customers in that market is a direct loss of potential revenue.

On a self-managed segmented architecture this interval includes all the steps needed to provision the necessary hardware, replicate index configurations across the environment, rebuild synonym lists, re-tune ranking rules and complete a full test cycle for the new locale prior to promoting it into production. On a managed blended index the interval is simply an exercise in configuring the system; extend the field mapping, select the analyzer, configure the locale specific query rules. For companies who operate in seasonal markets (Golden Week, Singles' Day, etc.), the difference between launching a new language market in days versus months is not measured in how long it takes to get engineers to deploy the product, but in whether you are able to capitalize on the revenue opportunities presented during those seasonal windows.

Algolia currently serves over 17,000 organizations in 150+ countries who conduct over 30 billion searches per week – or 1.75 trillion searches per year. This enormous volume of searches provide the data needed to determine the performance metrics for Algolia's services. As mentioned earlier, Algolia has developed two retrieval methods (language-aware and hybrid) which run in parallel and will not affect the end user's latency. Additionally, all of its services including language-aware tokenization, locale-based ranking, and hybrid retrieval are performed within the exact same timeframe as a standard language-specific search. Therefore, it has already made the decision on the engineering aspects of whether you can achieve interactive-latency multilingual search – it's already there. Now your only choice is how you wish to get there: either build/maintain the necessary infrastructure, or configure it.

6. Strategic relevance governance: the institutional control plane

In order to determine whether or not there is a strategic relevance governance issue regarding institutional control planes; The 'relevance problem' (finding products on a multilingual marketplace) turns into a 'relevance maintenance problem' (keeping products relevant to consumers through time) once a product(s) has/has been successfully launched onto a multilingual marketplace (in this case twelve languages). A 'relevance maintenance problem' is defined as a continuous cycle of trying to ensure that consumers can find products they want to purchase as the size and complexity of a marketplace grows and evolves. In other words, a relevance maintenance problem occurs every time a change is made to a product name in the English language version of a marketplace (for instance), it is most likely that the product names in all other languages will not be updated simultaneously. When a weight is added to promotional activities for a holiday such as Black Friday in the United States, it is very probable that the weights will be applied to the remaining eight locations that use the same promotional weights as well.

Furthermore, a feature that is added to enhance the quality of search results in one marketplace is not guaranteed to exist in another marketplace's schema. Thus, if a "silent error" occurs during a drift, it may either completely disregard the new attribute used in the ranking calculation, or it may alert the end user to the fact that the marketplace encountered an error. As a result, although the cumulative effect of each individual "drift" is small, the overall number of configuration changes to a marketplace over the course of twelve months of releasing product(s) across multiple languages creates a marketplace environment in which the relevance quality of Market Five and Market Nine are significantly lower than that of Market One, due solely to the success of transmitting configuration changes from the former to the latter and the failure of the latter to recognize and incorporate the transmitted configuration changes.

There are three pieces to the Governance Framework to support three different types of failure:

1. Index Contract – The Index Contract describes, maintains, and formally captures the Schema, Analyzers, and Rank Attributes needed in each index for each language. This means there is one location where you will always be able to find the proper configuration for a multilingual index. Therefore, if a configuration issue exists, it will be recognized at deploy time as a configuration issue and not as a loss of relevance or as an unknown occurrence weeks later.
2. Query Routing Governance – Query Routing Governance determines how BCP 47 locale signals are moved from a user's session to a search query, and what fields and/or Query Rules are run on the locale signal. Using the locale tag allows one to identify the applicable rules and rank signals for that tag without having to analyze the application code.
3. Relevance Measurement Parity – Relevance Measurement Parity is the promise to measure relevance in the same manner (click-throughs, conversions, no results) in all languages markets using the same review cycle. If Relevance Measurement Parity is not established, then relevance will diminish in lower visibility markets and remain unreported. A higher No Results rate for Japanese Language over a 3 month period of time is not a language-based issue; it is a governance-based issue.

{
  "conditions": [{
    "context": "ja-JP"
  }],
  "consequence": {
    "params": {
      "optionalFilters": ["market_popularity.ja_JP:>100"],
      "ruleContexts": ["ja-JP"]
    },
    "promote": [{
      "objectID": "seasonal_jp_featured",
      "position": 0
    }]
  }
}

Beginning with the first phase of this process (the Unspoken/Implicit Phase) the Unspoken/Implicit Phase typically doesn't require any specific action and is typically unspoken or implicit. As a result of the nature of our typical development processes, configuration exists but has never been formally documented or validated, and relevance is only measured in the primary language market.

The next phase (Managed) will differ from the first phase of this process in several ways. There will be version control of the contracts, automated validation of the contracts will exist for every deployed configuration and per-locale metrics will be evaluated on a basis that allows individuals to track them.

The final phase (Optimized) will occur when the loop closes and click and conversion data from all markets is used to re-rank the synonyms and expand the synonyms using the same review processes as any other configuration changes -- NOT a separate process for multilingual configurations, just the process itself.

Achieving this state will largely depend upon creating institutional norms within engineering organizations. The technical foundations required already exist.

It is typically the institutional norms that recognize multilingual relevance as an engineering concern common to all teams, rather than each locale team seeing it individually as its own engineering concern that is missing.

Those norms cannot take hold without a unified configuration surface. Each language implementation is being managed by different teams at different times and against no singular schema; therefore they never will converge. Governance is not a conceptual issue in this regard; the lack of a singular point to enforce such convergence is the reason.

Algolia fit: strategic relevance governance

Arguably the best part of the design of Algolia is the place where all of this resides. Index settings live in the API which means they can be audited at any time. Query rules will fire deterministically at 6 months exactly as they do at launch; the "ja-JP" rule will fire for "ja-JP," period. No manual verification needs to occur to see if it was fired or not. The analytics layer adds a tag to every query with the locale tag applied so you can measure relevance parity across locales versus assuming relevance parity exists. This is a platform for global experiences that enforces consistency, explainability, and performance.

7. Conclusion: global discovery is a relevance problem

Algolia's proprietary NeuralSearch technology incorporates both vector-based natural language processing and keyword matching in a single API. It does not supplant translation or language models — it merely takes their outputs and turns those outputs into something usable for users regardless of location. Algolia resolves the ranking and retrieval of results based on configured signals but the intent behind a user's search is still resolved by the NLP Layer. Algolia enforces consistency, explainability, and performance. Global discovery is a relevance problem. The linguistic layer has been solved. The retrieval layer is where the heavy lifting is done and where outcomes are resolved.

---

Request a demo of Algolia’s AI Search solution at algolia.com/demorequest or read more at algolia.com/blog/ai/