To add fast and feature-rich third-party APIs into any codebase, developers need to work within an ecosystem designed to include these self-sufficient APIs.

Developers also need to rely on technologies like GraphQL with Apollo Client and React to help them manage the variety of functionality and data sources.

In most situations, GraphQL is a godsend. As a flexible data layer, it helps centralize and standardize the data exchanges between the back and front ends.

However, requiring all data exchanges to flow through GraphQL can sometimes hinder an external API from reaching its full potential as a critical functional component. In our case, we wanted to integrate a cloud-based search API that performed substantially better than GraphQL.

The problem we solve in this article is how you can use an external API on its own terms, side-by-side with GraphQL — that is, without going through GraphQL.

Why we decided to bypass GraphQL with Apollo Client and How we achieved interoperability

We divide this article into two parts:

• The explanation for why we decided to bypass GraphQL to allow our front end to make direct calls to an external API’s cloud service
• The code we used to build the interoperability between the external API and GraphQL with Apollo Client

Feel free to jump to the code. But the backstory may help those faced with the same question: How do you combine GraphQL with self-sufficient APIs that provide their own set of data?

Why GraphQL? Why an external search API?

To say our online service is data-intensive is an understatement. Openbase helps developers find open-source libraries and packages perfectly tailored to their needs. They can search for and compare open-source packages with powerful metrics and user reviews. This information comes from many sources, including Github and our own databases.

And there are other concerns:

• Weekly downloads
• Maintenance
• Bundle size
• Categorization

Using GraphQL with Apollo Client

Openbase is a GraphQL shop. We access our data with @apollo/client for data-fetching and use React for rendering. A number of our React components are built using @apollo/client#readFragment to reference data from our GraphQL api. This makes our components tightly coupled to @apollo/client and GraphQL and less re-usable with other data sources.

Example:

Code for PackageTile.tsx:

import { useApolloClient } from '@apollo/client'
import React, { FC } from 'react'

// Generated using GraphQL Code Generator
import { AlgoliaProductFragment, AlgoliaProductFragmentDoc } from '@openbase/graphql'

interface PackageTileProps {
	packageId: string
}

const PackageTile: FC<PackageTileProps> = ({ packageId }) => {
	const apolloClient = useApolloClient()

	const packageFragment = apolloClient.readFragment<AlgoliaProductFragment>({
		fragment: AlgoliaProductFragmentDoc,
		id: `Package:${packageId}`
	})

	return (
		<div>
			<div>{packageFragment.name}</div>
			<div>{packageFragment.description}</div>
			<div>{packageFragment.bundleSize.gzipSizeInBytes}</div>
		</div>
	)
}

Building search with a third-party, cloud-based search API

With all that data, search is therefore essential to our business. Our developer-users start with a search bar. But they do more than that — they categorize, filter, and surf through multiple results and category pages, all of which comes with the external search API. On top of that, our product provides ratings and recommendations and allows users to personalize their results to their favorite languages. Much of this additional functionality comes directly from an external Search API’s results, in combination with our own datasources accessible via GraphQL.

The two options for combining GraphQL and an external search API

There are two mutually-exclusive approaches to integrating an external API into a GraphQL front-end layer:

1. Use the API on the front end as a secondary datasource alongside @apollo/client
2. Use the API on the back end behind our GraphQL API, and maintain a single front-end datasource

Option 1 – the reason for bypassing GraphQL

The external search API, provided by Algolia, is fast, displaying search results instantaneously as the user types. Putting Algolia behind our GraphQL API would bottleneck the search and slow it down considerably.

1. Algolia’s API shows an average search speed of _32ms across over 300,000 requests (at the time of this writing)
2. Our GraphQL API requests can take anywhere between 100-200ms sometimes

This delay is untenable in comparison.

To get that additional speed, which is critical to our product, we really needed to call the API directly.

Option 2 – the reasons to put the external API behind GraphQL

We had two concerns with bypassing GraphQL:

• Caching. Existing components built upon GraphQL’s cache aren’t aware of records retrieved through our search API.
• Schema-less. The search API’s records are schema-less. Its indexes are flat and contain the minimum set of attributes necessary for search, display, and ranking. While necessary for speed, relevance, and scalability, a flat, schema-less set of data does not align easily with the object-types from our GraphQL schema, which our components are built upon.

Additionally, putting search functionality behind GraphQL that integrates the search results into the front-end components just works out of GraphQL’s box: the front end just makes requests to GraphQL, retrieving the same object-types, and the developer experience stays consistent.

The decision

We were therefore faced with the following choice: either we go for the easier solution (plug the API into GraphQL) or we work a little harder to get the full functionality we expect from the API.

We decided with little hesitation to offer our users instantaneous search results as they type.

How we integrated the external API into our GraphQL front-end components

Algolia uses indexes that are not part of our GraphQL layer. We’ll be referring to algolia from this point forward.

Our GraphQL client is called @apollo/client.

Based on the above considerations, we’ve introduced an algolia-to-@apollo/client interoperability layer so that we are able to:

1. Consume the API directly on the frontend (to stay fast)
2. Use our existing react components built against GraphQL to maintain a developer and user experience that is consistent with the rest of our application

At a high level, the interoperability works like this:

1. Records are queried from the algolia-indices
2. The data from the records are written to our @apollo/client cache using custom mapper functions and GraphQL fragments generated from @graphql-codegen/cli
3. All of the components work out of the box reading data using GraphQL from @apollo/client‘s cache

Here is how we’ve built our indices to achieve this (built with typescript):

Code for algoliaToGraphQL.tsx:

import type { ObjectWithObjectID, SearchOptions, SearchResponse } from '@algolia/client-search'
import type { RequestOptions } from '@algolia/transporter'
import type { ApolloClient, DocumentNode } from '@apollo/client'
import type { SearchClient, SearchIndex } from 'algoliasearch'

// Narrowly type our indices for intellisense
export type AlgoliaTypedIndex<TRecord> = Omit<SearchIndex, 'search'> & {
	search: (query: string) => Readonly<Promise<SearchResponse<TRecord & ObjectWithObjectID>>>
}

// Define the schema for configuring the custom mappings
export interface AlgoliaToGraphQLFieldConfig<
	TIndex extends ObjectWithObjectID,
	TFragment extends Record<string, any>
> {
	__typename: string
	fragment: DocumentNode
	fragmentName?: string
	id?: (value: TIndex) => string
	write: (value: TIndex) => MaybePromise<Maybe<TFragment>>
}

// Define how the custom mapping will type-translate to narrowly-typed indices
export interface AlgoliaToGraphQLFields {
	[indexName: string]: AlgoliaToGraphQLFieldConfig<any, any>
}

export type AlgoliaToGraphQLResult<T extends AlgoliaToGraphQLFields> = {
	[P in keyof T]: T[P] extends AlgoliaToGraphQLFieldConfig<infer I, any>
		? AlgoliaTypedIndex<I>
		: never
}

// Create algolia-indices that will inject hits data to our `@apollo/client` cache
const createIndex = async <
	TIndex extends ObjectWithObjectID,
	TFragment extends Record<string, any>
>(
	algolia: SearchClient,
	apollo: ApolloClient<object>,
	name: string,
	config: AlgoliaToGraphQLFieldConfig<TIndex, TFragment>
): Promise<AlgoliaTypedIndex<TIndex>> => {
	const {
		__typename,
		fragment,
		fragmentName,
		id = (value) => `${__typename}:${value.objectID}`,
		write,
	} = config

	const writeFragment = async (value: TIndex): Promise<void> => {
		const fragmentData = await write(value)

		!!fragmentData && apollo.writeFragment<TFragment>({
			fragment,
			fragmentName,
			data: { __typename, ...fragmentData },
			id: id(value),
		})
	}

	const index = algolia.initIndex(name) as AlgoliaTypedIndex<TIndex>

	return {
		...index,
		// Override search to write everything into cache.
		async search(query, opts) {
			const result = await index.search(query, opts)

			await Promise.all(result.hits.map(async (hit) => writeFragment(hit)))

			return result
		},
	}
}

// Generate all of the new algolia indices from a config
export const algoliaToGraphQL = async <T extends AlgoliaToGraphQLFields>(
	algolia: SearchClient,
	apollo: ApolloClient<object>,
	config: T
): Promise<AlgoliaToGraphQLResult<T>> => {
	const indices = await Promise.all(
		Object.entries(config).map(async ([indexName, fieldConfig]) => {
			const index = await createIndex(algolia, apollo, indexName, fieldConfig)

			return [indexName, index] as readonly [string, AlgoliaTypedIndex<any>]
		})
	)

	return indices.reduce(
		(acc, [indexName, index]) => ({ ...acc, [indexName]: index }),
		{} as AlgoliaToGraphQLResult<T>
	)
}

Once we have the means to inject algoliasearch data to our @apollo/client cache, we simply type the shape of the Algolia record, define the GraphQL fragment to be written, map the record to the fragment, and build the new indices.

Code for types.ts:

import type { ObjectWithObjectID } from '@algolia/client-search'

// Types for our algolia records (for demonstration only)

interface BaseAlgoliaTypes {
	AlgoliaProduct: {
		name: string // String!
		description?: Maybe<string> // String
		bundleSize?: Maybe<number> // Int
		starRating: number // Float!
	}
}

export type AlgoliaTypes = {
	[P in keyof BaseAlgoliaTypes]: BaseAlgoliaTypes[P] & ObjectWithObjectID
}

Code for our GraphQL fragments for mapping:

fragment AlgoliaProduct on Package {
id
name
description
bundleSize {
gzipSizeInBytes
}
starRating
}

Code for indices.ts:

import type { ApolloClient } from '@apollo/client'
import type { SearchClient } from 'algoliasearch'
import { algoliaToGraphQL, AlgoliaToGraphQLFieldConfig } from './algoliaToGraphQL'
import type { AlgoliaTypes } from './types'

// Generated using GraphQL Code Generator
import { AlgoliaProductFragment, AlgoliaProductFragmentDoc } from '@openbase/graphql'

// Map algolia package records to our `AlgoliaProduct` fragment in GraphQL
const packages: AlgoliaToGraphQLFieldConfig<
	AlgoliaTypes['AlgoliaProduct'],
	AlgoliaProductFragment
> = {
	__typename: 'Package',
	fragment: AlgoliaProductFragmentDoc,
	write(value: AlgoliaTypes['AlgoliaProduct']): AlgoliaProductFragment {
		return {
			id: value.objectID,
			name: value.name,
			description: value.description,
			bundleSize: {
				gzipSizeInBytes: value.bundleSize,
			},
			starRating: value.starRating,
		}
	},
}

// Create all of our indices
export const getIndices = (algolia: SearchClient, apollo: ApolloClient<object>) => {
	return algoliaToGraphQL(algolia, apollo, {
		packages,
		// Different sortings using algolia virtual replicas
		packages_by_bundleSize_desc: packages,
		packages_by_starRating_desc: packages,
	})
}

Code for AlgoliaApolloProvider.tsx:

import { useApolloClient } from '@apollo/client'
import algolia from 'algoliasearch'
import React, {
	createContext,
	FC,
	ReactNode,
	useContext,
	useEffect,
	useState
} from 'react'
import { getIndices } from './indices'

export type AlgoliaApolloContextIndex = InferFromPromise<ReturnType<typeof getIndices>>

const AlgoliaApolloContext = createContext<Maybe<AlgoliaApolloContextIndex>>(null)

export interface AlgoliaApolloProviderProps {
	children?: ReactNode
}

// Wrap your application with this, to be able to use the interoperability anywhere
export const AlgoliaApolloProvider: FC<AlgoliaApolloProviderProps> = ({
	children,
}) => {
	const [index, setIndex] = useState<Maybe<AlgoliaApolloContextIndex>>(null)

	const apollo = useApolloClient()

	useEffect(() => {
		const newClient = algolia(process.env.ALGOLIA_APP_ID, process.env.ALGOLIA_API_KEY)

		getIndices(client, apollo).then((newIndices) => setIndex(newIndices))
	}, [apollo])

	return (
		<AlgoliaApolloContext.Provider value={index}>
			{children}
		</AlgoliaApolloContext.Provider>
	)
}

// Hook to use in any component that needs to search algolia
export const useAlgoliaIndex = () => useContext(AlgoliaApolloContext)

From this point on, whenever a search is made to Algolia, components that use @apollo/client‘s cache will work out of the box:

import { NextPage } from 'next'
import React from 'react'
import { useAlgoliaIndex } from './AlgoliaApolloProvider'

const Page: NextPage = () => {
	const index = useAlgoliaIndex()
	const [hits, setHits] = useState<any[]>([])

	useEffect(() => {
		index?.packages
			.search('react', { length: 20, offset: 0 })
			.then((results) => setHits(results.hits.slice()))
	}, [index])

	return (
		<div>
			{!!hits && hits.map((hit) => (
				<PackageTile key={hit.objectID} packageId={hit.objectID} />
			))}
		</div>
	)
}

export default Page

With this, we achieved the goals we’ve set for a fast search, with a developer and user experience that is consistent with the rest of Openbase. The API for our modified indices are the same as the non-modified indices from algoliasearch, and our components can stay unmodified — specific to our single @apollo/client data-source.

Coming up with a high-quality and relevant set of Github repos is an infinite challenge. Algolia’s algoliasearch eliminates a significant complexity for Openbase to build an experience that allows for users to find the best open-source tooling to fit their needs. The approach your GraphQL shop takes with integrating Algolia or any other API may depend on your own requirements and conditions.