GraphQL search and indexing with Algolia

Widely used on the web, GraphQL brings many advantages to developers. On the front end, it reduces roundtrips and enables better client-side caching. On the back-end, it simplifies the multiplicity of microservices by producing self-describing schema-based APIs. We’ll see how to enhance GraphQL search functionality in this article.

Algolia shares some of the same core values as GraphQL, such as providing a performant and streamlined way to access data and valuing a great developer experience. For this reason, when developers combine Algolia with GraphQL, they are given a rich set of tools to build a best-in-class search experience.

To see how these two technologies work together, we’ve created a fictional company that offers online flight price comparisons (flight-tickets.io). Flight-tickets.io’s system first leverages GraphQL to improve Algolia’s back-end indexing process, and then it integrates Algolia InstantSearch to provide a unified front-end programming environment.

Index rich data into Algolia with GraphQL

The company’s Algolia index exposes tickets prices that come from dozens of external providers. Without GraphQL, the ETL implementation to index all the providers’ data into the Algolia index was complex–it required a worker for each data source.

Here’s the flow:

As you can see, there were three steps: Pull data from multiple providers, transform it, and then push it to Algolia. The complexity of this process rapidly increased as the number of data sources and API types (REST, gRPC, …) grew, leading to slower integration and support from the engineering team for new providers.

There were stability issues as well:

each worker had to make multiple calls to multiple APIs for each data provider
each provider exposed data with its own protocol: REST, gRPC, or proprietary
custom error management was unique to each provider

Adding GraphQL-based indexing

The solution was to rewrite the ETL by leveraging GraphQL, as follows:

Now, the pull and transformation stages were managed (or unified) by one GraphQL process. By building a new GraphQL internal API, relying on GraphQL Mesh, the team added a single data layer between the data providers and the ETL.

Essentially, the ETL pulls or subscribes to data changes from the internal GraphQL API, which exposes multiple data sources that are queried in a unified way, as follows:

query getTicketsForProvider($provider: Provider!, $from: Date!, $to: Date!, $cursor: Cursor) {
  tickets(provider: $provider, from: $from, to: $to, cursor: $cursor) {
    edges {
      node {
        provider {
          id
          name
        }
        prices {
          ...PriceByLocationAndTime
        }
        # ...
      }
      pageInfo {
        cursor {
          before
          after
        }
      }
    }
  }
}

The data is uniform and includes error handling. Here’s a typical GraphQL error response body containing data and errors fields:

{
  "data": {
     “tickets”: []
  },
  "errors": [
    {
      "message": "[SuperFlight API]: unavailable",
      "locations": [
        {
          "line": 2,
          "column": 3
        }
      ],
      "extensions": {
        "code": "TICKETS_GRAPHQL_API_SUPERFLIGHT_ERROR",
        "exception": {
          "stacktrace": [
            "......",
          ]
        }
      }
    }
  ]
}

Finally, besides network latency optimization, GraphQL brings real performance optimizations on the provider’s data-fetching side. GraphQL API implementation is highly asynchronous by design: each data field is resolved in parallel, resulting in better performances.

Side-note:
For Algolia’s Shopify extension, we leverage Shopify’s GraphQL API to index Shopify data into Algolia indices. Shopify’s GraphQL API not only brings the same advantages already discussed, but it also goes further:

More generous and flexible rate-limiting, compared to the Shopify REST API

Advanced pagination patterns

We see a 28% increase in average indexing speed and a 150% increase in indexing peak capacity of Shopify data after migrating from the Shopify REST API to the Shopify GraphQL API.

GraphQL search with Instant Search

Now that they’ve unified with GraphQL, the airline company wanted to revamp their search UI to use a GraphQL back-end search for SEO and data enrichment reasons, backed by Algolia. While we always recommend front-end search (that is, to go directly to Algolia’s servers for maximum, instantaneous speed), in the case of reservations systems like reserving airlines tickets, it is sensible to take an extra trip back to the back-end server to fetch real-time data. As for SEO, you can often combine the back and front ends to enhance your company’s Google SEO.

Algolia GraphQL search

The first step was to expose a GraphQL search query on the existing public GraphQL API. To achieve this, the back-end team used a tool called algolia-graphql-schema that generated GraphQL types from a given Algolia index.

The flightsSearch Algolia index contained objects like the following:

{
  "provider_id": 12312312,
  "flight_company_name": "Bryan'air",
  "inbound": "2021-09-13T16:39:22.396Z",
  "outbound": "2021-19-13T16:39:22.396Z",
  "price": 1140.00
  // ...
}

They added the following script to the package.json:

// ...
"scripts": {
  "generate:graphql": "algolia-graphql-schema"
},
// …

And ran the script:

$ npm link
$ npm run generate:graphql

> algolia-graphql-demo-server@1.0.0 generate:graphql
> algolia-graphql-schema

Analyzing flightsSearch index (that might take a few seconds...)
flightsSearch.graphql created!

The following flightsSearch.graphql is generated:

type AlgoliaResultObject {
  provider_id: String!
  flight_company_name: String!
  inbound: Date!
  outbound: Date!
  price: Int!
  # ...
}

type SearchResultsEdge {
  cursor: Int!
  node: AlgoliaResultObject
}

type SearchResults {
  edges: [SearchResultsEdge!]!
  totalCount: Int!
}

input SearchInput {
  query: String
  similarQuery: String
  sumOrFiltersScores: Boolean
  filters: String
  page: Int
  hitsPerPage: Int
  offset: Int
  length: Int
  attributesToHighlight: [String]
  attributesToSnippet: [String]
  attributesToRetrieve: [String]
  # ...
}

type Query {
  # ...
  search_flights($input: SearchInput!, $after: String): [SearchResults!]!
}

Once they integrated the generated search types definition in the existing GraphQL API types and implemented the corresponding resolvers that called the Algolia search API, the back-end team gave the front-end team the green light to build the new Search UI using Algolia InstantSearch.

Algolia InstantSearch GraphQL integration

The front-end team was now able to use the search_flights($input: SearchInput!, $after: String) GraphQL Query to search for flights tickets. Fortunately, Algolia has documented how to use InstantSearch on the back end, so the setup was swift.

The front-end team implemented a custom GraphQL search client for their InstantSearch.

import { SearchClient } from 'instantsearch.js'

const query = `
  Search($input: SearchInput!, $after: String) {
    search_flights(input: $input, before: $after) {
      edges {
        cursor
        node {
          provider_id
          flight_company_name
          inbound
          outbound
          price
          # ...

        }
      }
      totalCount
    }
  }
`;

const HITS_PER_PAGE = 20

const transformResponse = ({ query, page }) => (response) =>
  response.data?.search_flights.edges.reduce(
    (acc, edge) => {
      acc.results[0].hits.push(edge.nodes);
      acc.results[0].nbHits += 1
    },
    {
      results: [
        {
            hits: [],
            page,
            nbHits: response.data?.search_flights.edges.length || 0,
            nbPages: response.data?.search_flights.totalCount / HITS_PER_PAGE,
            query: query || "",
            exhaustiveNbHits: false,
            hitsPerPage: HITS_PER_PAGE,
            processingTimeMS: 0,
            params: ""
          }
      ],
    }
  );

const searchClient: SearchClient = {
  searchForFacetValues: (() => {}) as any,
  search(requests) {
    const request = requests[0]
    return fetch("https://api.flightickets.io/graphql", {
      method: "post",
      headers: {
        "Content-Type": "application/json",
      },
      body: JSON.stringify({
        query,
        variables: { input: { ...request } },
      }),
    }).then(transformResponse({ query: request.query, page: request.page }));
  },
};

Note: The search UI code targets a single Algolia index through a GraphQL API.

The custom hits widget benefits from the TypeScript search results types generated from the GraphQL search API, as seen here:

// TypeScript types generated from the GraphQL API using GraphQL code generator
import { AlgoliaResultObject } from './graphql/generated'

const structuredResults = connectHits(({ hits, widgetParams }) => {
  const results = hits as AlgoliaResultObject[];
  // results.flight_company_name is autocompleted by TypeScript generated types

  const { container } = widgetParams;

  if (
    results
  ) {
    // Render the result
    // ...
    return;
  }

  // Render no results
  // ...
});

They’re ready to go. They just need to integrate Algolia into the GraphQL API as well as the front-end’s InstantSearch configuration:

On the API side, they need to call Algolia with the JavaScript client
On the InstantSearch side, they need to provide the searchClient and a custom hits widget to InstantSearch

Conclusion

By combining Algolia and GraphQL, flight-tickets.io improved their overall user and developer experience.

First, by building a faster, more stable, and rich indexing pipeline leveraging GraphQL to consume and unify data from dozens of providers:

A unique data language and error management
Some performance improvements thanks to GraphQL parallel architecture
Less complexity leading to an improved developer experience

Finally, by leveraging GraphQL and building a back-end search, the front-end team perfectly integrated the search into the front-end stack with Algolia InstantSearch.

Have any feedback on this post? I’d love to hear from you: @whereischarly

ABOUT THE AUTHOR