Search by Algolia
Feature Spotlight: Query Rules
product

Feature Spotlight: Query Rules

You’re running an ecommerce site for an electronics retailer, and you’re seeing in your analytics that users keep ...

Jaden Baptista

Technical Writer

An introduction to transformer models in neural networks and machine learning
ai

An introduction to transformer models in neural networks and machine learning

What do OpenAI and DeepMind have in common? Give up? These innovative organizations both utilize technology known as transformer models ...

Vincent Caruana

Sr. SEO Web Digital Marketing Manager

What’s the secret of online merchandise management? Giving store merchandisers the right tools
e-commerce

What’s the secret of online merchandise management? Giving store merchandisers the right tools

As a successful in-store boutique manager in 1994, you might have had your merchandisers adorn your street-facing storefront ...

Catherine Dee

Search and Discovery writer

New features and capabilities in Algolia InstantSearch
engineering

New features and capabilities in Algolia InstantSearch

At Algolia, our business is more than search and discovery, it’s the continuous improvement of site search. If you ...

Haroen Viaene

JavaScript Library Developer

Feature Spotlight: Analytics
product

Feature Spotlight: Analytics

Analytics brings math and data into the otherwise very subjective world of ecommerce. It helps companies quantify how well their ...

Jaden Baptista

Technical Writer

What is clustering?
ai

What is clustering?

Amid all the momentous developments in the generative AI data space, are you a data scientist struggling to make sense ...

Vincent Caruana

Sr. SEO Web Digital Marketing Manager

What is a vector database?
product

What is a vector database?

Fashion ideas for guest aunt informal summer wedding Funny movie to get my bored high-schoolers off their addictive gaming ...

Vincent Caruana

Sr. SEO Web Digital Marketing Manager

Unlock the power of image-based recommendation with Algolia’s LookingSimilar
engineering

Unlock the power of image-based recommendation with Algolia’s LookingSimilar

Imagine you're visiting an online art gallery and a specific painting catches your eye. You'd like to find ...

Raed Chammam

Senior Software Engineer

Empowering Change: Algolia's Global Giving Days Impact Report
algolia

Empowering Change: Algolia's Global Giving Days Impact Report

At Algolia, our commitment to making a positive impact extends far beyond the digital landscape. We believe in the power ...

Amy Ciba

Senior Manager, People Success

Retail personalization: Give your ecommerce customers the tailored shopping experiences they expect and deserve
e-commerce

Retail personalization: Give your ecommerce customers the tailored shopping experiences they expect and deserve

In today’s post-pandemic-yet-still-super-competitive retail landscape, gaining, keeping, and converting ecommerce customers is no easy ...

Vincent Caruana

Sr. SEO Web Digital Marketing Manager

Algolia x eTail | A busy few days in Boston
algolia

Algolia x eTail | A busy few days in Boston

There are few atmospheres as unique as that of a conference exhibit hall: the air always filled with an indescribable ...

Marissa Wharton

Marketing Content Manager

What are vectors and how do they apply to machine learning?
ai

What are vectors and how do they apply to machine learning?

To consider the question of what vectors are, it helps to be a mathematician, or at least someone who’s ...

Catherine Dee

Search and Discovery writer

Why imports are important in JS
engineering

Why imports are important in JS

My first foray into programming was writing Python on a Raspberry Pi to flicker some LED lights — it wasn’t ...

Jaden Baptista

Technical Writer

What is ecommerce? The complete guide
e-commerce

What is ecommerce? The complete guide

How well do you know the world of modern ecommerce?  With retail ecommerce sales having exceeded $5.7 trillion worldwide ...

Vincent Caruana

Sr. SEO Web Digital Marketing Manager

Data is king: The role of data capture and integrity in embracing AI
ai

Data is king: The role of data capture and integrity in embracing AI

In a world of artificial intelligence (AI), data serves as the foundation for machine learning (ML) models to identify trends ...

Alexandra Anghel

Director of AI Engineering

What are data privacy and data security? Why are they  critical for an organization?
product

What are data privacy and data security? Why are they critical for an organization?

Imagine you’re a leading healthcare provider that performs extensive data collection as part of your patient management. You’re ...

Catherine Dee

Search and Discovery writer

Achieving digital excellence: Algolia's insights from the GDS Retail Digital Summit
e-commerce

Achieving digital excellence: Algolia's insights from the GDS Retail Digital Summit

In an era where customer experience reigns supreme, achieving digital excellence is a worthy goal for retail leaders. But what ...

Marissa Wharton

Marketing Content Manager

AI at scale: Managing ML models over time & across use cases
ai

AI at scale: Managing ML models over time & across use cases

Just a few years ago it would have required considerable resources to build a new AI service from scratch. Of ...

Benoit Perrot

VP, Engineering

Looking for something?

facebookfacebooklinkedinlinkedintwittertwittermailmail

As a SaaS company, our software and infrastructure must stay ahead of the competition. And it must be reliable: server & API availability are as important as innovation. In other words, while we need to deliver product updates and new features as quickly and as often as possible, we also need to build resilient software by testing everything – which takes away time from innovating.

Experience has taught us how to balance these competing needs: test in production.

Why do resilient testing in production?

Resilient testing in production is the process of deploying new code in production, and “testing” it using the production traffic directly instead of running an exhaustive test suite. This is risky. You usually want to avoid it, but as your application grows, it becomes impossible to test everything.

In the Algolia engine, we currently have more than twenty query parameters. If they were all boolean flags, testing all cases would require a million tests: twenty parameters, with two possible values per each, that’s 2^20 possibilities.

When it comes to the time that testing takes, there are three things to take into account:

  • Time to write the tests
  • Time to maintain the tests 
  • Time to run the tests

Writing a million tests is already a long process, but once they’re there, these tests become part of the project. As such, they require maintenance like the rest of the source code, so each software iteration now requires more work.

Now let’s imagine that your team is large enough to write and maintain those tests, you still need to run them. If each test only takes ten milliseconds, it adds up to 2h45m to run. Any update in the code now takes 2h45m to validate.

Our customers buy our product for the existing set of features, but also the upcoming services. They expect us to release new features regularly so we can help them grow, to be innovative and resilient. Therefore, we need to be efficient.

When we implement a new feature, we only write tests to validate the implementation and to detect obvious corner cases. To thoroughly verify the functionality, we decided to put a deployment setup in place that allows us to test in production with minimal impact for our customers in case of a bug. This way, we can deliver new features on time.

Real-life example

Let’s take a concrete example: we recently rewrote the highlighting feature of Algolia. If we wanted to test it exhaustively, this would mean combining the one million tests needed for all parameters with all the possible Unicode characters (over a million). This adds up to over a billion tests. If each test only took a millisecond to run, this test suite itself would take 11 days to complete.

We had to find a better solution. Therefore, instead of maintaining billions of tests and significantly slowing down our release process, we decided to test in production.

Our main concern was to avoid impacting customers, so we defined what we needed to put such a setup in place:

  • A progressive deployment process
  • A way to retry on a healthy infrastructure
  • Proactive issue detection

Anyone who subscribes to Algolia from our website gets access to a three-node cluster. Each node owns 100% of the data and can serve queries independently, which provides a resilient setup. When you consider an average availability of 95% with bare metal servers, this setup allows us to provide 99.987% availability. For your Algolia service to be completely down, each server would need to be down. The probability is, therefore, 5% of 5% of 5%, or 0.0125% of potential downtime.

Now, even with this setup, a software bug could cause a service outage. For that reason, we have put in place a progressive deployment process of the search engine which spans across three days. This partial deploy gives us enough time to detect issues.

If there’s a bug in the new version, our API clients take over with their retry strategy. If they target a broken node, they transparently retry with another one until they get a successful response. From the end user’s perspective, the issue is invisible.

When we deployed the rewrite of the highlighting feature, we detected a normalization issue. The goal was for it to convert text in a canonical form to simplify comparison between different inputs. Normalized forms are usually shorter or as long as the original form, and that’s what the highlighting code was expecting. However, there are some characters for which the normalized form is longer: the ß (German sharp S) is normalized into ss. During the rewrite, we added runtime pre-conditions checks to ensure the normalized size was shorter or equal, and the code behaved as expected. This is what revealed the bug.

When we deployed the new version on the first node, it immediately stopped responding to queries that were generating a longer normalized form. However, thanks to our API client’s retry strategy, none of our customers noticed it. On our end, our monitoring system triggered an alert, and we quickly reverted the change to stabilize the engine. It left us all the necessary time to properly understand the issue, write a proper fix, and the corresponding tests.

A suitable setup

There are three crucial elements that you need when you want to test in production:

  • A replicated infrastructure
  • A resilient software
  • A safe deployment strategy

Replicated infrastructure

Nowadays, configuring a replicated infrastructure is straightforward: all cloud providers offer load balancers in front of multiple VMs. In our case, we handle the replication of the search engine data at the cluster level, with each node owning 100% of the data. This way, nodes can reply to queries independently.

Resilient software

This part highly depends on the software you’re building. In the Algolia engine, we have plenty of health checks within the code to verify the pre-conditions of a function and expected state. When we come to an unexpected state, the engine stops processing to avoid returning corrupted data. It forces the API clients to retry on a different node transparently.

Safe deployment strategy

Last and not least, the deployment process’s main goal here is to release new versions while controlling risks progressively.

At Algolia, we’ve split our infrastructure into four environments: testing, staging, production, and safe. Each environment has a different SLA:

  • Testing contains clusters that we use internally. Breaking it only impacts the company.
  • Staging contains search clusters of public-facing projects that are maintained by Algolians.
  • Production contains the clusters of our customers.
  • Safe contains the clusters of our premium SLA customers.

In our deployment strategy, we leverage those different environments to mitigate risk. This means we first deploy on environments with the least impact for our customers.

In addition to the different environments, we also leveraged the replication factor of three, and created a deployment process in twelve steps:

  1. We deploy on all three nodes of the testing clusters.
  2. We deploy on the first node of the staging clusters, then on the first node of the production clusters, then on the first node of safe clusters.
  3. We wait one day.
  4. We deploy on the second node of the staging clusters, then on the second node of the production clusters, then on the second node of safe clusters.
  5. We wait one day.
  6. We deploy on the third node of the staging clusters, then on the third node of the production clusters, then on the third node of safe clusters.

The first step allows us to detect issues in the code that handles the distributed system and the interactions between nodes within a cluster.

Then, deploying on each node one by one lets us assess whether a cluster can support two different versions at the same time and if the code is stable.

The rationale behind waiting for a whole day before deploying on the next environment is that it leaves enough time to detect performance issues, data corruption, or time-based issues. Once we reach this step, we’ve detected most of the problems. The rest of the progressive deployment is here to help us identify uncaught breaking changes.

Whenever we detect a bug, we immediately revert the new version. This way, we can set the service back into a stable state, and this gives us enough time to fix the issue and add the corresponding tests correctly.

With this approach, our test suite is driven by customer usage. This makes it particularly efficient and allows us to release a new version of the engine every week to sustain the needs of our customers, even with such a large codebase.

Good is better than perfect

The startup ecosystem is challenging. Small teams need to find efficient strategies to create better products than big companies with large teams.

Regularly releasing new features is as important as having a stable product with a great user experience that answers the demand of customers. This need for efficiency is the reason why we must find a middle ground between writing enough tests to have good feature coverage, and being able to release often.

You should always be careful with adding more tests because they are time-consuming: it takes time to write them, maintain them, and run them. So how do you know when to write tests? The ninety-ninety rule applies: it’s usually straightforward to test 90% of a feature, and the last 10% takes as much time. That’s why it’s crucial to handle the remaining 10% based on customer usage, instead of pushing for exhaustive coverage.

To mitigate the risk, make sure you take the time to design a piece of software and an infrastructure that supports tests in production, with the least possible impact on customers.

About the author
Xavier Grand

Engineering Manager

Take ownership of your career and our business

View jobs
Take ownership of your career and our business

Recommended Articles

Powered byAlgolia Algolia Recommend

Good API Documentation Is Not About Choosing the Right Tool
engineering

Maxime Locqueville

DX Engineering Manager

Algolia's top 10 tips to achieve highly relevant search results
product

Julien Lemoine

Co-founder & former CTO at Algolia

Introducing our new navigation
product

Craig Williams

Director of Product Design & Research