We don’t always write about the data or content side of search, which we call indexing. To be honest, we take indexing for granted. We say, “send us your data”, and then, once that’s out of the way, you can start focusing on the critical parts – building powerful customer-facing search interfaces, increasing conversions, customer engagement, and catalog discovery. 

Not that there’s anything wrong with primarily writing about how to build a great business-value search UI/UX. But by not talking about how Algolia indexes your data, we skip over a step that lies at the heart of what we offer and creates just as much value for your business as the search interfaces that you build.

The value of search indexing is that it can transform any and all back-office data and file types into searchable data, enabling fast and relevant search on any device, globally, for any user – customers, partners, and employees. Once this multitude of data is placed into a single touch point between the back and front ends, you can leverage Algolia’s fully-featured Search & AI in all facets of your business, to handle any use case, internally or online.

Indexing as a middle layer between the front and back ends

In this respect, we can think of Indexing as on a par with Search and Recommend. 

• For starters, everything you do with our search and recommendations products references back to our fast indexing and cloud infrastructure

• The speed of our search is based on how we index content and store your data on our servers

• The way we manage, behind the scenes, the sending and storage of your data is secure, robust, reliable, and capable of acting as a middle layer between all systems and all potential users, and in any B2C, B2B, and system-to-system use case

Most companies have one or more general purpose databases that store business-critical content like product data, sales history, and customer, inventory, and financial information. Algolia is not where this data is stored originally. Algolia is not a general purpose storage database. Nor is it an application that manages a specific business functionality, like customer relations or product and inventory management.

We don’t expect every sale to be sent to Algolia. We don’t provide that kind of search. That’s the territory of a purpose-specific POS system. But Algolia does have a role in the back office: it is built to interface with every source of truth, to provide a fast and relevant searchable dataset fed by any back-office system. Its cloud infrastructure provides a reliable and secure searchable data layer that lies between the back-office data source and any front-end that serves data to users. 

Algolia’s power is to help surface the content in these back-office systems, to lead users to the original information. For example, Algolia can be used to store just enough information about products – price, image, description, amount available, and popularity, to help customers buy or employees to manage. In this use case, clicking on a product should take the user back to a product page, whose detail comes directly from the original source of data. 

Instantaneous search, fast indexing

To serve all front-end search needs, it’s important that Algolia provide a solid indexing foundation. We will soon be publishing a series of articles on “headless search”, where we’ll discuss how our API-first indexing decouples the back from the front end, to create this middle layer of searchable data. Be the first to find out about these articles, as well as other current and future content by subscribing to our newsletter. 

Let’s break down the differences between Algolia’s search and indexing options and offerings.

Search

We’ve designed and optimized Algolia to provide instantaneous search results, so that there is no delay between typing a letter and getting an immediate response. Our SLA is that we ensure search availability 99.99% of the time. Here are the some details of our software’s commitments and offerings:

• Real-time search
• Relevance 
• Advanced features
• AI relevance
• Front end UI/UX libraries
• Business benefits – increased conversions and customer engagement

Indexing

Behind every search lies a simple indexing framework – a single call to send any data from any system in a completely flexible format.

• Flexibility – we can structure any data for search purposes
• Size – we offer unlimited data capacity when hosted on several instances in our cloud architecture
• Secure
• Globally available
• Reliable – SLA, always available

Search vs Indexing

• Search takes priority – whenever a search is requested, the search is treated before any indexing
• Indexing is structured to return the fastest, informative, and display-ready search results
• Good indexing practices can process data in under 1 second or up to 5 to 10 seconds

Why do we give search priority over indexing? Two main reasons:

• To match user expectations that they get instant results as they type their queries
• To not miss a single search request, because every search is a potential conversion 

A typical search and indexing use case

The typical search and indexing process goes as follows: 

1. Users search a company’s product catalog and ultimately want to convert (buy).
2. A user buys a product online or in a store. 
3. The online or in-store POS app processes the transaction and sends the data to the back end servers.
4. The back-office sales software (POS) stores the transaction.
5. On a regular basis – depending on the company’s use case – the back-office updates the data in Algolia. That is, the back office POS system sends new products, or updated product data, like availability and popularity data, to Algolia, thereby affecting the new content that is displayed to the users as they keep entering search queries. 

Note that the re-indexing needs to be regular, but not for every sale. A single sale does not change the catalog. And it is rare that a significant change occurs in the course of a single day, let alone minutes. Price, availability, promotion campaigns, and custom rankings like popularity take time to adjust.

Thus, the most important concern for our customers is to ensure that all searches are a success and instantaneous, and that the back-end updates are regular and responsive enough so that significant changes are taken into account in a timely manner. Usually, this means every 30 minutes, or even 1 hour, or overnight. Every company chooses its best timeframe for its various use cases. 

There are, however, some active timeframes, where a company needs 10-second updates, or even instant updates. Let’s discuss these unique use cases.

Advanced search and indexing use cases

Let’s look at a definition before getting started.

Real-time can mean:

• Actual updates that are taken into account (in the search) as soon as they happen in the real world. This is a stock market scenario.
• Updates that are taken into account every X seconds after they happen, where X is typically a low number.

The following use cases will require one of these.

Fast-Indexing in a crisis

The crisis starts with a container ship lodged into the river bed of the Suez Canal. It’s not only incapable of delivering its own goods, it’s also blocking other container ships and therefore a large portion of the world’s supply of goods. On the sales side, ecommerce companies all over the world have to rethink their stock immediately, to avoid out-of-stock orders. They also have to find replacement items and adjust their promotional campaigns, to keep their online catalogs as engaging as before the crisis. While their IT departments re-index the online catalogs to remove the blocked items, their merchandisers set out to procure the replacement items and rethink their promotions. 

On the other side of the equation, wholesale suppliers have to match the new demand by removing the blocked items from their B2B catalogs and push different products to their buyers. 

Indexing shows up in three parts: 

• Removing the blocked items
• Adding the new items
• Adjusting their promotional campaigns

Fast indexing in a sudden surge of buying and selling (e.g., Black Friday)

Another case requiring a fast indexing response occurs in the 5 weeks between Black Friday and the end of the Christmas season. In this scenario, companies need to constantly monitor their stocks and keep ahead of the market, to ensure they can fulfill every order and respond to their customer’s needs with the best and most important items with every search and promotion. In this case, they monitor fairly often – some companies multiple times throughout the day.

Fast Indexing in a constantly changing online catalog (e.g., a marketplace)

In this scenario, we have a large marketplace with independent vendors who are constantly updating their inventories. For the most part, these vendors do not own a full stock of items; instead, they own one of each product. Therefore, one sale can make a product unavailable, and so they need to remove objects as soon as a sale is made. Some marketplaces allow users to put a temporary hold on objects that are added to a wish list. 

Every marketplace website comes up with a strategy to manage this unpredictable and fast-moving flux of out-of-stock and product holds. Here are some strategies:

• The UI explicitly reminds users of availability (displaying, “only 1 item left” next to the item).
• If a user puts an item in their cart, the UI allows the vendor to put the item on hold.
• The UI explicitly tells other users if an item is on hold. They can add it to a wish list and be notified if the item becomes free or unavailable.

eBay adds an additional complication, where the buying and selling of items is joined by an auction and bids that change prices on the fly. The way you would handle this kind of real-time updating depends on the UI. However, this case comes very close to needing real-time data processing on both search and indexing. Let’s look closer at this last use case.

Real-time Indexing in a real-time market

The situation here is that every time a vendor makes a change to a product – especially price – they want users to see this change (and any associated calculations) immediately. As such, it is no different than a typical stock market tool, where you can see prices change like flickering lights. Consider an automated trading tool that buys and sells stocks as stock prices change. Any delay impacts the competitiveness of the system and could impact large amounts of money.

One can put into question if any B2C scenario has such a need for real-time indexing. Do prices change that much, that quickly? Do vast amounts of people and systems buy and sell goods based on the smallest price change? 

Perhaps B2B has a better case for needing instantaneous price changes to be displayed. It is quite possible suppliers out-bid each other in real-time. 

In any bidding universe – we can add eBay to this – it is critical to have the fastest indexing response times.

This is where we get to the heart of fast indexing. Is any search engine’s indexing going to be as fast as its search functionality? That is, while a search engine provider contractually commits to displaying instant results as users type, it does not commit in the same way to “instant” or near real-time indexing. This is because, in order to achieve real-time search (search in milliseconds), you must index data in a certain way that inevitably takes time (1 to 10 seconds, depending on the size of the index and the number of updates in the indexing request).

Algolia values a faster-than-database search (milliseconds) at the expense of a slower-than-database indexing process (seconds). And as seen in the advanced use cases above, even in a crisis or constantly changing inventory / pricing environment, the speed of our indexing engine is reliable, responsive, and exceeds expectations. 

Next readings

This article has presented a high-level overview of standard and advanced indexing use cases. Our next article walks you through indexing best practices and the implementation details of a standard indexing process. That’s followed up by an article on how to optimize indexing in advanced use cases.

Our remaining articles will provide front & back end code for some of the advanced indexing use cases we discussed, starting with real-time pricing.

To find out how Algolia’s powerful indexing and cloud infrastructure can transform your digital strategy, sign up for free and see it for yourself. Or get a customized demo from our search experts today.