The challenge

The Times has been a leading voice for decades with its ongoing mission to provide clarity and knowledge to its audience while holding people of power accountable through its journalism. 

As more and more people look online for their news, it's clear that the growth of media is in digital. Recognizing this, The Times has continued to evolve and has risen to meet the needs of its readers with its digital mission of providing a high-quality journalistic service which readers find valuable and worth subscribing to. 

With the demand and consumption of online media growing, equipping digital editors and journalists with fast, efficient and flexible tools to produce compelling, highquality stories, as well as build, curate and publish the collection of that content is essential - especially in a world where myriad distractions are only a click away.

Building daily online editions requires a new approach

In order to build a great online edition each day that The Times's journalists can be proud of, the team recognized early on that taking the print version and simply replicating that online would not meet their own standards, nor their growing online readership. 

While internal tools usually never get the right attention or resources within most organizations, The Times's digital team knew that in order to better serve its readers and online subscribers, it had to meet the day-to-day demands of building a valuable, online edition that represents the very best journalistic content that The Times was known for. After proving the value through an initial prototype built in a week, the digital team worked with Times management to prioritize building a technology application called Edition Builder. 

"Internal tools never get enough attention. Building them can be a thankless, unseen task that's rarely prioritised, as it is not immediately obvious how to tie investment in them back to the bottom line. It can be too easy to jump on the customer-facing speedboat and jet away from them, leaving a wake of stopgaps that can often leave you with tightly coupled intricate workarounds that are enormously difficult to pick apart. This was the case at The Times for the last four years. While we spent a long time discussing and designing how editions might work on our readers' screens, we didn't place any focus on how to make building these experiences manageable for our editors." Matt Taylor ("Building Editions at The Times", January 2019) 

The goal of Edition Builder was simple: help publishers more quickly and intuitively build, edit, and publish tomorrow's news for The Times and The Sunday Times from the best, most relevant content across hundreds of articles written and submitted each day. 

However, in the initial user testing of Edition Builder, the production team was frustrated with the system and were struggling to take the most basic actions to build the different pages of the online edition and site, causing the overall edition build process to take too long. 

This was largely due to a monolithic content management system (CMS) along with an internally-built search tool that indexed articles from the CMS within Edition Builder that was unable to quickly and consistently find and retrieve relevant, submitted articles. 

In addition to poor search results, this prior search solution only enabled in-memory searching and therefore required more data to be brought into a user's browser session of Edition Builder which significantly impacted performance. Finally, users could only narrow search results by filtering between two dates. 

Constructing the index within Edition Builder took as long as 3 seconds for each article within every 24 hour period of content searched. Users couldn't afford to wait several seconds every time they needed new articles, so the team limited the search window to only the most recent 72 hours. To make matters worse, to retrieve new articles, a user had to refresh the local index, causing further delay. 

Essentially, search in Edition Builder was unworkable and too slow. Due to this poor user experience and slow performance, the digital team made the decision that the prior tool had to be replaced. 

With someone from the publishing team coming in at 6:00 pm each evening to start going through 200-300 stories in order to create the story and set of pages for tomorrow's edition, it was critical that the new Edition Builder be enabled with a seamless, scalable and fast search experience. 

Realizing that this would not be a problem that could be solved, nor should be solved, by internal development, the digital team sought a Saas solution whose total cost of ownership (TCO) would be substantially less than the engineering time to build and maintain its own solution.

The solution

Delighting publishers and digital teams with fast, responsive search

During this same time, The Times had recently partnered with Algolia to power search on their external site to replace a legacy Solr cluster that was deemed too complex and difficult to maintain. 

After this decision was made and the implementation had begun, the Digital team saw an opportunity to also use Algolia to power search for Edition Builder and solve some of the performance and functional challenges that had plagued the earlier version. 

Tearing out the old system and a rapid implementation of Algolia within Edition Builder took just a few days. Within Algolia, the team then configured ranking rules and added filtering and faceting to meet all their edition building needs including being able to filter by topic, journalist, any date (vs. only date range in the prior system).

The next step was then to place the new Edition Builder release in front of their journalists to test. The team saw substantial improvements immediately. Feedback from the production team was that the experience was fast and responsive, providing the relevant results quickly to allow them to be more productive to focus on the content they were developing, instead of being slowed down by the technology. 

Through Algolia, instead of a local in-browser index, every query goes to an index that Algolia hosts, resulting in more recent and relevant results that are returned almost instantly as users type each letter into the search box, including articles that were just indexed seconds ago.

While about 80% of the time, the production team will search and find articles submitted in the past 24 hours, 20% of the time they need to search across the entire library of over 250,000 published and unpublished articles representing 4 gigabytes of data indexed by Algolia to find the right article. With Algolia, results could easily be refined by users who could easily filter by topic, journalist, publication, and version.

"Algolia is a breeze to work with. With Algolia, our editorial team has seen significant productivity improvements when building the daily online edition of The Times and weekly edition of The Sunday Times, with search being 300 - 500 times faster than our prior solution."

Finally, the digital team can now easily view and adjust how search relevance and result rankings were determined with full transparency within Algolia to continuously improve the Edition Builder experience. Unlike a 'black box' style search system that would typically require dedicated engineering resources, the team can see how the ranking affected the results for a given search term, and then simply and securely adjust relevance accordingly based on rules they managed within Algolia, even on a per-query basis (e.g., boosting important articles to the top of search results based on attributes such as headline, author and filename.)

The result

Delivering value while providing times savings

The latest version of Edition Builder with Algolia is seen as a real success within The Times. The users evaluating "Edition Builder" have recognized that Edition Builder is far faster - and in some cases 50-100x faster than the previous workflow to perform the same task - with Algolia search as a key component for this success. 

Based on analysis from the team, with this latest edition, approximately 3 hours and 30 minutes have been saved each week across 36 users with Algolia search results 300 - 500 times faster than the prior solution. 

Looking forward to the future, the team sees this solution much easier to maintain and index and allows The Times digital team to scale for future use cases. In addition, it plans to take further advantage of Algolia's search and discovery platform on its external site.

"By having Edition Builder's search powered by Algolia, we're able to save over three and a half hours each week compared to our prior solution and now able to focus on building the best edition of The Times for our readers each day."