Search by Algolia
Easily integrate Algolia into native apps with FlutterFlow
engineering

Easily integrate Algolia into native apps with FlutterFlow

Algolia's advanced search capabilities pair seamlessly with iOS or Android Apps when using FlutterFlow. App development and search design ...

Chuck Meyer

Sr. Developer Relations Engineer

Algolia's search propels 1,000s of retailers to Black Friday success
e-commerce

Algolia's search propels 1,000s of retailers to Black Friday success

In the midst of the Black Friday shopping frenzy, Algolia soared to new heights, setting new records and delivering an ...

Bernadette Nixon

Chief Executive Officer and Board Member at Algolia

Generative AI’s impact on the ecommerce industry
ai

Generative AI’s impact on the ecommerce industry

When was your last online shopping trip, and how did it go? For consumers, it’s becoming arguably tougher to ...

Vincent Caruana

Senior Digital Marketing Manager, SEO

What’s the average ecommerce conversion rate and how does yours compare?
e-commerce

What’s the average ecommerce conversion rate and how does yours compare?

Have you put your blood, sweat, and tears into perfecting your online store, only to see your conversion rates stuck ...

Vincent Caruana

Senior Digital Marketing Manager, SEO

What are AI chatbots, how do they work, and how have they impacted ecommerce?
ai

What are AI chatbots, how do they work, and how have they impacted ecommerce?

“Hello, how can I help you today?”  This has to be the most tired, but nevertheless tried-and-true ...

Catherine Dee

Search and Discovery writer

Algolia named a leader in IDC MarketScape
algolia

Algolia named a leader in IDC MarketScape

We are proud to announce that Algolia was named a leader in the IDC Marketscape in the Worldwide General-Purpose ...

John Stewart

VP Corporate Marketing

Mastering the channel shift: How leading distributors provide excellent online buying experiences
e-commerce

Mastering the channel shift: How leading distributors provide excellent online buying experiences

Twice a year, B2B Online brings together America’s leading manufacturers and distributors to uncover learnings and industry trends. This ...

Jack Moberger

Director, Sales Enablement & B2B Practice Leader

Large language models (LLMs) vs generative AI: what’s the difference?
ai

Large language models (LLMs) vs generative AI: what’s the difference?

Generative AI and large language models (LLMs). These two cutting-edge AI technologies sound like totally different, incomparable things. One ...

Catherine Dee

Search and Discovery writer

What is generative AI and how does it work?
ai

What is generative AI and how does it work?

ChatGPT, Bing, Bard, YouChat, DALL-E, Jasper…chances are good you’re leveraging some version of generative artificial intelligence on ...

Catherine Dee

Search and Discovery writer

Feature Spotlight: Query Suggestions
product

Feature Spotlight: Query Suggestions

Your users are spoiled. They’re used to Google’s refined and convenient search interface, so they have high expectations ...

Jaden Baptista

Technical Writer

What does it take to build and train a large language model? An introduction
ai

What does it take to build and train a large language model? An introduction

Imagine if, as your final exam for a computer science class, you had to create a real-world large language ...

Vincent Caruana

Sr. SEO Web Digital Marketing Manager

The pros and cons of AI language models
ai

The pros and cons of AI language models

What do you think of the OpenAI ChatGPT app and AI language models? There’s lots going on: GPT-3 ...

Catherine Dee

Search and Discovery writer

How AI is transforming merchandising from reactive to proactive
e-commerce

How AI is transforming merchandising from reactive to proactive

In the fast-paced and dynamic realm of digital merchandising, being reactive to customer trends has been the norm. In ...

Lorna Rivera

Staff User Researcher

Top examples of some of the best large language models out there
ai

Top examples of some of the best large language models out there

You’re at a dinner party when the conversation takes a computer-science-y turn. Have you tried ChatGPT? What ...

Vincent Caruana

Sr. SEO Web Digital Marketing Manager

What are large language models?
ai

What are large language models?

It’s the era of Big Data, and super-sized language models are the latest stars. When it comes to ...

Catherine Dee

Search and Discovery writer

Mobile search done right: Common pitfalls and best practices
ux

Mobile search done right: Common pitfalls and best practices

Did you know that 86% of the global population uses a smartphone? The 7 billion devices connected to the Internet ...

Alexandre Collin

Staff SME Business & Optimization - UI/UX

Cloud Native meetup: Observability & Sustainability
engineering

Cloud Native meetup: Observability & Sustainability

The Cloud Native Foundation is known for being the organization behind Kubernetes and many other Cloud Native tools. To foster ...

Tim Carry

Algolia DocSearch is now free for all docs sites
product

Algolia DocSearch is now free for all docs sites

TL;DR Revamp your technical documentation search experience with DocSearch! Previously only available to open-source projects, we're excited ...

Shane Afsar

Senior Engineering Manager

Looking for something?

facebookfacebooklinkedinlinkedintwittertwittermailmail

Fast indexing ensures that search results contain the most up-to-date information in the most timely manner. The meaning of “timely” depends on every company’s particular business activity. But as mentioned in our previous article on indexing best practices, and repeated below, fast indexing comes out of the box – that is, there’s nothing you need to do to optimize the indexing process, it’s already fast enough for most use cases. 

Optimizing an already fast indexing process might seem unnecessary. But in some situations, optimizing adds speed to the standard indexing process. We see this in a handful of advanced use cases where data needs to be updated even faster than usual, sometimes in real-time. 

We’ll discuss the following advanced use cases:

  • High-frequency data changes
  • Crisis
  • Black Friday or other high-activity events
  • Real-time indexing, scenario 1: booking a hotel, reservations & rental availability
  • Real-time indexing, scenario 2: bidding and stock markets

*Just a general note on indexing. For all use cases, whether standard or advanced, you’ll always want to perform a “batch indexing process” that updates data in small batches, incrementally, and per attribute

High-frequency data changes

In this scenario, it’s about how you prioritize some changes over others. We’ll call this “selective updating”. For example, you decide to send some updates now (like price) and others later (like description). This lowers the number of changes in each batch,  shortening the frequency of your batching process (e.g., every 5 minutes to every 1 minute).

Another technique is to use “non-delete indexing”, where you set an “is-unavailable” attribute on a record instead of deleting it. Deleting records costs more in terms of performance than merely updating attributes. But it’s a trade-off. While it’s faster to change the value of an attribute, it’s always better to reduce the size of an index via deleting, which makes it faster to index. Thus, the best practice is to do the techniques together: make attribute-changes every 5 minutes and delete those records every 30 minutes.

Crisis

A crisis situation happens when a factory doesn’t deliver a supply of goods. Or a ship gets stuck in the Suez canal, blocking the world’s supply of goods. In these scenarios, the online business has to immediately remove the unavailable items from its website and replace them with another set of products. They may also have to rethink their promotions. 

There’s actually no reason to do anything different from a standard indexing process. However, if the crisis goes on for too long, or requires a lot of changes in a short time, it’s important to avoid overloading the indexing. The best approach then is to categorize the changes:

  • Remove out-of-stock items immediately. You have two options here: delete the items from the index or use an “out-of-stock” boolean, filtering out records whose “out-of-stock” attribute is false. As mentioned in the previous section, the best approach is to change the attributes every 5 minutes and delete the flagged records every 30 minutes.
  • Update the new items as they arrive, in batched updates separately from the other 2 categories.
  • Change the promotions separately, in batched updates separately from the other 2 categories.

Black Friday or other high-activity events

Black Friday combines the scenarios of high-frequent changes and “crisis”, so you’ll want to follow the suggestions of those scenarios. The difference here is that on Black Friday, the high-activity period could last longer than a crisis, or create sudden spikes of activity – but at least you can plan for the event in advance, which makes it easier to manage.

Real-time indexing, scenario 1: booking a hotel, reservations & rental availability

Users expect search results to have the most up-to-date, accurate information. Technically, this means that they want to see changes in the back office to appear in their search results immediately. Businesses want this as well: outdated information can negatively impact profits and customer trust.

Example: booking a hotel. If a hotel appears available in the search results, users expect that it has the same status and price when they click for more detail. But what if the hotel is booked between the query and the click? You can manage this gracefully with a friendly front-end UX (see below). But you can also mitigate the problem with an additional indexing strategy, often used in real-time systems-level programming that relies on a second, smaller index.

You create a tiny index on the Algolia server that collects updates, up to 1000 records. The front-end code merges this tiny index with your main index on every search, removing results (e.g., out-of-stock items) or updating information (e.g., price) on the fly. Here’s the algorithm:

  • The Main index contains all product data, as normal.
  • The Tiny index gets updated with each change as it occurs, not yet the main index. This tiny index should contain no more than 1000 records. 
  • For every query, the front end merges the two indexes. There are two possibilities: 
    • The Tiny index contains one attribute per record, the object id. If the record exists in the results (matched on object id), it’s removed from the results.
    • The Tiny index contains 2 attributes (object id, price). The object id is used to match records in the results and the price attribute to overwrite the price information in the results.
  • Meanwhile, on the back end, the main index gets updated regularly with the data in the tiny index (following the standard batch-update techniques described previously). The tiny index is then zeroed out, ready to receive new updates.

Notes:

  • The client-side merge should not impact performance because client-side merging is simple, fast, and contains only a small amount of records.
  • Pagination: Removing results will affect the pagination, because the removed items will create gaps in the list of results. Thus, some search result pages will have less items than others. To solve this, place a banner or a promotion in the gap. But the best solution is to use infinite scrolling or “load more” logic if you need to remove items often.  

Real-time indexing, scenario 2: bidding and stock markets

Here, user expectations and business needs are more stringent: this use case requires that changes in the back-office show up immediately in the search results. We see this in stock market trading or bidding applications, where prices can change every second or even millisecond. If this scenario is yours, contact Algolia and discuss the different advanced settings you can adjust on your application, engine, and data that can maximize your indexing performance. While engine-level changes should rarely be done, in exceptional circumstances, they can shorten indexing times.

Other considerations

Perceived performance – Front end UI/UX solutions  

One important aspect to performance is perceived performance. We won’t cover UI/UX best practices here, but want to acknowledge the importance of building a front end that gives the feeling of high performance. This is not about creating a false impression, it’s about communicating to the user that there is a (reasonable) waiting time. A friendly progress bar is an example of this: it asks the user nicely to wait, and if it’s not too long, people are ok with that. There are many equally effective ways to manage performance on the UI.

Out-of-the-box performance

As promised, a word about what we mean by “out-of-the-box high performance”. Our indexing comes with the following technologies:

  • A search engine using advanced indexing techniques
  • High-performant bare-metal servers configured for performance 
  • A globally available cluster-based cloud infrastructure, with low latency and server redundancy (i.e., no server downtime)
  • An API with a retry method to ensure (contractually) 99.99% availability 

Next readings

Our first article on indexing presented a high-level overview of standard and advanced indexing use cases. Our next article walked you through indexing best practices and the implementation details of a standard indexing process. This article discussed how to optimize indexing in advanced use cases.

Now it’s time to help you build solutions. Our remaining articles will provide front & back end code for some of the advanced indexing use cases we discussed, starting with real-time pricing. 

To get started with indexing, you can upload your data for free, or get a customized demo from our search experts today.

About the author
Peter Villani

Sr. Tech & Business Writer

linkedinmediumtwitter

Recommended Articles

Powered byAlgolia Algolia Recommend

An Exploration of Search and Indexing: Fast Indexing Scenarios
product

Peter Villani

Sr. Tech & Business Writer

Search indexing best practices for top performance (with code samples)
engineering

Peter Villani

Sr. Tech & Business Writer

Inside the Algolia Engine Part 1 — Indexing vs. Search
engineering

Julien Lemoine

Co-founder & former CTO at Algolia