Search by Algolia
Haystack EU 2023: Learnings and reflections from our team
ai

Haystack EU 2023: Learnings and reflections from our team

If you have built search experiences, you know creating a great search experience is a never ending process: the data ...

Paul-Louis Nech

Senior ML Engineer

What is k-means clustering? An introduction
product

What is k-means clustering? An introduction

Just as with a school kid who’s left unsupervised when their teacher steps outside to deal with a distraction ...

Catherine Dee

Search and Discovery writer

Feature Spotlight: Synonyms
product

Feature Spotlight: Synonyms

Back in May 2014, we added support for synonyms inside Algolia. We took our time to really nail the details ...

Jaden Baptista

Technical Writer

Feature Spotlight: Query Rules
product

Feature Spotlight: Query Rules

You’re running an ecommerce site for an electronics retailer, and you’re seeing in your analytics that users keep ...

Jaden Baptista

Technical Writer

An introduction to transformer models in neural networks and machine learning
ai

An introduction to transformer models in neural networks and machine learning

What do OpenAI and DeepMind have in common? Give up? These innovative organizations both utilize technology known as transformer models ...

Vincent Caruana

Sr. SEO Web Digital Marketing Manager

What’s the secret of online merchandise management? Giving store merchandisers the right tools
e-commerce

What’s the secret of online merchandise management? Giving store merchandisers the right tools

As a successful in-store boutique manager in 1994, you might have had your merchandisers adorn your street-facing storefront ...

Catherine Dee

Search and Discovery writer

New features and capabilities in Algolia InstantSearch
engineering

New features and capabilities in Algolia InstantSearch

At Algolia, our business is more than search and discovery, it’s the continuous improvement of site search. If you ...

Haroen Viaene

JavaScript Library Developer

Feature Spotlight: Analytics
product

Feature Spotlight: Analytics

Analytics brings math and data into the otherwise very subjective world of ecommerce. It helps companies quantify how well their ...

Jaden Baptista

Technical Writer

What is clustering?
ai

What is clustering?

Amid all the momentous developments in the generative AI data space, are you a data scientist struggling to make sense ...

Vincent Caruana

Sr. SEO Web Digital Marketing Manager

What is a vector database?
product

What is a vector database?

Fashion ideas for guest aunt informal summer wedding Funny movie to get my bored high-schoolers off their addictive gaming ...

Vincent Caruana

Sr. SEO Web Digital Marketing Manager

Unlock the power of image-based recommendation with Algolia’s LookingSimilar
engineering

Unlock the power of image-based recommendation with Algolia’s LookingSimilar

Imagine you're visiting an online art gallery and a specific painting catches your eye. You'd like to find ...

Raed Chammam

Senior Software Engineer

Empowering Change: Algolia's Global Giving Days Impact Report
algolia

Empowering Change: Algolia's Global Giving Days Impact Report

At Algolia, our commitment to making a positive impact extends far beyond the digital landscape. We believe in the power ...

Amy Ciba

Senior Manager, People Success

Retail personalization: Give your ecommerce customers the tailored shopping experiences they expect and deserve
e-commerce

Retail personalization: Give your ecommerce customers the tailored shopping experiences they expect and deserve

In today’s post-pandemic-yet-still-super-competitive retail landscape, gaining, keeping, and converting ecommerce customers is no easy ...

Vincent Caruana

Sr. SEO Web Digital Marketing Manager

Algolia x eTail | A busy few days in Boston
algolia

Algolia x eTail | A busy few days in Boston

There are few atmospheres as unique as that of a conference exhibit hall: the air always filled with an indescribable ...

Marissa Wharton

Marketing Content Manager

What are vectors and how do they apply to machine learning?
ai

What are vectors and how do they apply to machine learning?

To consider the question of what vectors are, it helps to be a mathematician, or at least someone who’s ...

Catherine Dee

Search and Discovery writer

Why imports are important in JS
engineering

Why imports are important in JS

My first foray into programming was writing Python on a Raspberry Pi to flicker some LED lights — it wasn’t ...

Jaden Baptista

Technical Writer

What is ecommerce? The complete guide
e-commerce

What is ecommerce? The complete guide

How well do you know the world of modern ecommerce?  With retail ecommerce sales having exceeded $5.7 trillion worldwide ...

Vincent Caruana

Sr. SEO Web Digital Marketing Manager

Data is king: The role of data capture and integrity in embracing AI
ai

Data is king: The role of data capture and integrity in embracing AI

In a world of artificial intelligence (AI), data serves as the foundation for machine learning (ML) models to identify trends ...

Alexandra Anghel

Director of AI Engineering

Looking for something?

facebookfacebooklinkedinlinkedintwittertwittermailmail

Application development is often reactive. We see the need, we deliver the solution as fast as possible. During this fast software cycle, we gather requirements and implement them as soon as they appear. I’m not talking about quick and dirty. I’m referring to using the best RAD practices – rapid application development.

The RAD cycle is as follows: you implement great core features (MVP-style), relying on years of experience to create maintainable code. But over time, several things occur: requirements change, more code gets written, and the codebase starts to rebel against your intuitively brilliant but perhaps not fully robust architecture. So you start refactoring. Also, you discover that technology changes, offering new ways to make your code simpler, cleaner, and more powerful.

Enter game changer React Hooks. And, a fast growing business that requires you to rewrite your application with loads of new features.

Rewrite – from scratch. Life offers a second opportunity.

How React Hooks saved our administration application

Application development can also be pro(Re)active. Our administration application is data-intensive. Previously, many separate (and competing) components had managed their data independently – connecting, formatting, displaying, updating, etc..

The requirements of an Admin application

An Admin application is a good candidate for centralizing data handling. Administrators need to see the data as is, so the onscreen views usually match the structure of the underlying data. So, while our client-facing dashboard presents functional views for business users, an administrator needs to see user or client subscription information in a consistent and straightforward manner.

What we needed was a more scalable solution. Since we pull data from multiple sources – all accessible via one API with many endpoints – we wanted to centralize the common aspects of data handling. This not only gave us immediate benefits (better testing, caching, syncing, standard typing), it facilitated and simplified future data integrations.

A customized hook

We implemented a custom React hook called useData, which manages and therefore centralizes all data-retrieval API calls, data exchanges, type checking, caching, and other such data-based functionality. The caching alone enhanced user-facing speed enormously.

Equally important, the speed and centralization enabled our front-end developers to reuse their components and UI elements in different parts of the interface. Such reusability created a feature-rich, user-friendly UI/UX without front-end developers needing to maintain unique state information within each component. Lastly, under the hood, data reusability enabled a coherence in the models that drove the front-end functionality.

We will discuss front-end benefits of React hooks in future articles; this article is about how we served the front-end with a reliable and scalable layer of data handling.

How our useData hook centralized the process

We use different data sources, some more complex than others but all following the same JsonAPI specification. Additionally, they all have the same needs – a means to:

  • Retrieve data
  • Deserialize and format it
  • Validate its format
  • Perform error handling (data quality, network)
  • Synchronize with app refreshes and other data/workflows
  • Cache the data and keep it up to date

Enough talking, here’s our useData hook code:

import { useCallback } from 'react';
import { useQuery, useQueryClient } from 'react-query';
import { ZodObject, infer as Infer } from 'zod';
import { useApi } from 'hooks';
import { metaBuilder, MetaInstance } from 'models';

interface Options {
  forceCallApi?: boolean;
  preventGetData?: boolean;
}

interface ApiData<T> {
  data?: T;
  meta?: MetaInstance;
}

export interface DataResult<Output> {
  data?: Output;
  meta: any;
  loading: boolean;
  errors: Error[];
  refresh: () => Promise<void>;
}

export const useData = <Model extends ZodObject<any>, ModelType = Infer<Model>, Output extends ModelType = ModelType>(
  builder: (data: ModelType) => Output,
  url: string,
  { forceCallApi = false, preventGetData = false }: Options = {}
): DataResult<Output> => {
  const queryClient = useQueryClient();

  const { getData } = useApi(url);

  const getDataFromApi = useCallback(async (): Promise<ApiData<Output>> => {
    // here we get the data (and meta) using getData, and handle errors and various states
    return { data: builder(apiData), meta: metaBuilder(apiMeta) }
  }, [getData, builder, queryClient, url, forceCallApi]);

  const { data: getDataResult, isLoading, error } = useQuery<ApiData<Output>, Error>(
    [url, forceCallApi],
    getDataFromApi,
    { enabled: !preventGetData, cacheTime: forceCallApi ? 0 : Infinity }
  );

  const refresh = useCallback(async () => {
    await queryClient.refetchQueries([url, forceCallApi], {
      exact: true,
    });
  }, [queryClient, url, forceCallApi]);

  return {
    data: getDataResult?.data,
    meta: getDataResult?.meta,
    loading: isLoading,
    errors: ([error]).filter((error) => error !== null) as Error[],
    refresh,
  };
};

As you can see, this hook takes three parameters that, when combined, give us all the following functionalities:

  • A “builder” function that transforms and enhances the data for use by our components
  • The URL of the API endpoint that retrieves the data
  • Optional parameters. For example, to ignore cache or wait for some other data to be ready before calling the API

The result is that our components no longer need to manage all that. We’ve abstracted and encapsulated the complexity.

The useData hook returns some values we can use in our components:

  • Some states: loading and errors (if any)
  • The data (if any)
  • Meta information (if present – pagination information, for example)
  • A refresh function (to refresh the data by calling the API again)

Building the data

Let’s take a deeper look at what this code does and how we use it.

Schema validation with Zod

Getting the data is one thing. Ensuring that the data is correctly structured, or typed, is another. Complex data types require validation tools like yup or zod that enforce efficient and clean methods, and offer tools and error handling runtime errors based on faulty types. Our front end relies on strongly-typed data sets, so the validation stage is crucial for us.

We use zod. Zod is used to build a model of the data. For example, here’s what the model for our Application could look like:

import { object, string, number } from 'zod';

const Application = object({
  applicationId: string(),
  name: string(),
  ownerEmail: string(),
  planVersion: number(),
  planName: string(),
});

Then, to construct our builder function, we use in-house-built generic helpers on top of the zod model.This helper takes two parameters:

  • The model of our data (Application in our example above)
  • A transformer function that is used to enrich that model.

In our case, that transformer would look like this:

import { infer as Infer } from 'zod';

const transformer = (application: Infer<typeof Application>) => ({
  ...application,
  get plan() {
    return `${application.planName} v${application.planVersion}`;
  },
});

Another example of enrichment is if a model has a date: we usually want it to expose a javascript date rather than a string date.

We have 2 versions of that helper function (one for objects and one for arrays). Below is the first one:

import type { ZodType, TypeOf, infer as Infer } from 'zod';
import { SentryClient } from 'utils/sentry';

export const buildObjectModel = <
  Model extends ZodType<any>,
  ModelType = Infer<Model>,
  Output extends ModelType = ModelType
>(
  model: Model,
  transformer: (data: TypeOf<Model>) => Output
): ((data: ModelType) => Output) => {
  return (data: ModelType) => {
    const validation = model.safeParse(data);
    if (!validation.success) {
      SentryClient.sendError(validation.error, { extra: { data } });
      console.error('zod error:', validation.error, 'data object is:', data);
      return transformer(data);
    }
    return transformer(validation.data);
  };
};

The typed output by zod is very clean and looks like a typescript type that we would have written ourselves, with the addition that zod parses the JSON using our model. For safety, we use the safeParse method from zod, which allows us to send back the JSON “as is” in case of an error during the parsing step. We would also receive an error on our error tracking tool, Sentry.

With our example, our builder function would look like:

export const applicationBuilder = buildObjectModel(Application, transformer);
// and for the record, here is how to get the type output by this builder:
export type ApplicationModel = ReturnType<typeof applicationBuilder>;
// which looks like this in your code editor:
// type ApplicationModel = {
//   plan: string;
//   applicationId: string;
//   name: string;
//   ownerEmail: string;
//   planVersion: number;
//   planName: string;
// }

Calling the API

Internally, we use another custom hook useApi (less than 200 lines of code) to handle the GET/POST/PATCH/DELETE. In this hook, we use axios to call the backend API and perform all typical CRUD functionality. For example, on the read side, Axios deserializes the data we receive before it’s converted from the JSON API spec to a more classic JSON, and switching from snake_case to camelCase. It also handles any meta information we receive.

Also, from a process point of view, it manages request canceling and errors when calling the API.

Caching the data

At this point, we can summarize: the useApi hook gets the data, which is then passed through the builder to be validated and enriched; and the resulting data is cached using react-query.

We implemented react-query for caching the data on the front end, using the API endpoint URL as the cache key. React-query uses the useApi hook mentioned above to fetch, synchronize, update, and cache remote data, allowing us to leverage all these functionalities with a very small codebase.

All we have to do on top of that is implement react-query’s provider. To do so, we’ve constructed a small react component:

import { FC } from 'react';
import { QueryClient, QueryClientProvider, QueryClientProviderProps } from 'react-query';

export const queryClient = new QueryClient({
  defaultOptions: {
    queries: {
      refetchOnWindowFocus: false,
      refetchInterval: false,
      refetchIntervalInBackground: false,
      refetchOnMount: false,
      refetchOnReconnect: false,
      retry: false,
    },
  },
});

type IProps = Omit<QueryClientProviderProps, 'client'> & {
  client?: QueryClient;
};

export const GlobalContextProvider: FC<IProps> = ({
  children,
  client = queryClient,
  ...props
}) => (
  <QueryClientProvider {...props} client={client}>
    {children}
  </QueryClientProvider>
);

Most importantly, it manages our caching. We have many components that need the same data, so we wanted to avoid unnecessary network traffic to retrieve the same information. Performance is always key. And so is limiting potential errors performing unnecessary network calls. Now, with caching, if one component asks for data, our cache will store that data and give it to other components that ask for the same information. In the background, React-query of course ensures that the data in the cache is kept up to date.

To sum up, here’s an example of a component built using this useData hook and our Application model as defined above:

import { FC } from 'react';

interface ApplicationProps {
  applicationId: string;
}

export const ApplicationCard: FC<ApplicationProps> = ({ applicationId }) => {
  const { loading, data: application, errors } = useData(applicationBuilder, `/applications/${applicationId}`);

  return loading ? (
    <div>loading...</div>
  ) : errors.length > 0 ? (
    <div>{errors.map(error => (<div>{error}</div>))}</div>
  ) : (
    <div>
      <div>{application.applicationId}</div>
      <div>{application.ownerEmail}</div>
      <div>{application.name}</div>
      <div>{application.plan}</div>
    </div>
  );
};

As you can see, our useData hook lets us standardize the loading and errors states, encouraging us to write reusable components that handle those states. For example, we have reusable StateCard and StateContainer components. With the data now easily available, we can go about integrating those reusable components and focus exclusively on building a great front end experience – cleanly, fully-featured, and scalable.

About the author
Emmanuel Krebs

Sr. Software Engineer

githubtwitter

Recommended Articles

Powered byAlgolia Algolia Recommend

Part 4: Supercharging search for ecommerce solutions with Algolia and MongoDB — Frontend implementation and conclusion
engineering

Soma Osvay

Full Stack Engineer, Starschema

Adding trending recommendations to your existing e-commerce store
engineering

Ashley Huynh

Creating an omnibar with Autocomplete
engineering

Bryan Robinson

Senior Developer Relations Specialist