Serving One Billion JavaScript Library Downloads

This month, with the help of jsDelivr, we reached a milestone 1 billion downloads (that’s 26TB!) across all of our libraries. Since we now know how to deliver that many JavaScript libraries in a fast, robust and stable way, we thought we’d share our experience with the JavaScript community.

Algolia is developing multiple open source projects (like InstantSearch.js) to simplify the integration process of our search engine. When a developer wants to include one of our JavaScript libraries, she can use either:

• The npm package published on the npm registry
• A JavaScript file hosted on a CDN, to be used in a `<script>` tag

Providing those two ways allows us to target different use cases:

• Advanced JavaScript users familiar with module bundlers like webpack
• Beginner or intermediate JavaScript users used to copy/paste `<script>` tags in their html

Our goal with this strategy is to reduce the friction of using our service at every level of the developer experience.

Publishing on npm is rather easy: you just run `npm publish` in your directory and that’s it.

CDN delivery of JavaScript libraries

As for CDN delivery of our JavaScript libraries, an obvious option would be to find a commercial CDN provider and write a deployment script to automatically upload new versions to the CDN.

This also means someone would need to maintain this new system, configure the CDN, update the scripts, monitor uptime (if it fails, our users’ websites are down) and everything that comes with maintaining a live production system. Early on we decided that our main business was search and not delivering JavaScript libraries to the world, so we searched for solutions already providing this.

That’s when we found out about jsDelivr.

Delivering your libraries with jsDelivr

jsDelivr is a free CDN built exactly for this. It offers production quality of service and natively integrates with npm.

Our deployment process looks like this:

As you can see, the only manual action is to run `npm publish`. After that, the npm registry and jsDelivr handles everything.

Since the beginning, Algolia and jsDelivr have been in close relation. jsDelivr offers Algolia a great way to easily deliver JavaScript libraries to the world. And jsDelivr uses Algolia to provide JavaScript search on their website.

Dive into jsDelivr

jsDelivr was launched in 2012 and is currently developed and maintained by Prospect One, a company opened by the original founders of the project. Since the start, it was fully open source and focused on developers, including the development of an API for all public CDNs with the help of the community.

jsDelivr has a unique-among-public-CDNs infrastructure: it uses multiple CDN providers and load-balances between them based on performance and uptime, instead of using a single CDN provider.

At the moment, jsDelivr uses Cloudflare, Fastly and Stackpath for all global file delivery except for mainland China, which is exclusively served by Quantil, a company that has a large network of servers across all major Chinese cities. This is made possible by the ICP license held by jsDelivr.

In total, jsDelivr serves about 19 billion requests every month. That’s almost 500TB of bandwidth of small ~3kb js files — all that from 176 locations all around the world.

To see more detail on how jsDelivr works under the hood, check the infographic below that visually explains all the failover systems integrated into it.

Pro tip: websites do not need auto updates

jsDelivr has some great features to automatically provide updated version of packages to consumers. For example, you can use scripts like:

• https://cdn.jsdelivr.net/npm/instantsearch.js@latest would be always updated with any latest version
• https://cdn.jsdelivr.net/npm/instantsearch.js@2 would always be updated with the last minor version of instantsearch.js v2 (following semver)

This sounded very cool for early Algolia: we could update our client websites at any moment with auto updating, and we could advertise that they would be always up to date. Alas, WRONG!

In fact, this turned out to have many disadvantages:

• Extra consciousness about any line of code added to our libraries
• No way (when using /latest/) to make any substantial permanent breaking changes. For example, to release a completely revamped, modernized JavaScript client with a clear upgrade path was not feasible if people used /latest/: their website would auto update even before they could make changes to support new versions

Ultimately, when your website works and has no issues on Monday, there’s no need to push an update on Tuesday from our side. When there are new features and fixes you might want, we will communicate with you via our community forum or change log.

Nowadays, for production systems, we recommend using fixed versions (like https://cdn.jsdelivr.net/npm/instantsearch.js@2.1.0), and being aware of new updates when you need them.

Still, auto updating packages has their advantages. When creating jsFiddle or CodeSandbox examples for our users to show us bugs to be fixed, we use auto updating so that we never have to manually updates those templates.

The future of JavaScript delivery

JavaScript modules are a great addition to standardize the modularity aspect of JavaScript apps. If you want to learn more about them, read the modules chapter of Exploring ES6 book by Axel Rauschmayer.

This feature is now coming to Node.js and browsers, which means there will be adjustments to make to both our build and delivery strategy, so we can support new usages like `<script type=”module”></script>`.

We trust jsDelivr to help us make the right decisions and potentially build new features for JavaScript modules.

Big thanks to Dmitriy Akulov for his help with this post.