POLITICO: Delivering actionable information to subscribers through 8x faster search

Formed in September 2014 as part of a joint venture with German publisher Axel Springer SE, POLITICO Europe is a non-partisan politics and policy news organization that delivers high-quality European political analysis and content. Through premium politics and policy news coverage, POLITICO Pro is a customizable newsfeed and information platform that empowers thousands of policy experts and decision-makers from over 900 organizations.

In addition to detailed datasets, POLITICO Pro covers 7 policy areas: Agriculture and Food, Energy and Climate, Financial Services, Healthcare, Technology, Trade, and Mobility, as well as 5 cross-industry products: Brexit Transition, Sustainability, Cybersecurity and Data Protection, EU Budget and Competition and Industrial Policy. Subscribers include EU and national governments, corporations, trade associations, consultancies, law firms, and NGOs. 

Separate from the POLITICO website, the Pro platform pushes the boundaries of a traditional news source. By using technology to offer connected journalism, data, and tools, POLITICO is arming policy professionals with the insight they need to do more in a truly innovative way. For Pro users, staying informed about complex topics is essential to making an impact and remaining one step ahead. But it’s not always easy to know where to start. This is where search plays a vital role for POLITICO users — through direct searches and continuous content monitoring.

The challenge

POLITICO’s mission has always been “to win the audience” by providing accurate, objective “information to the right people at the right time so that they can act with confidence and speed.” 

Unfortunately, feedback from POLITICO Pro customers since 2018 had been that the Pro platform was not able to provide the information that customers were looking for through the original search experience on the platform. 

For example, Pro customers were using full sentences when searching (“e.g. What were the top 2019 data protection developments in France?”) and were rarely getting the results they expected. The POLITICO Pro sales team also offered similar feedback from their customers, and this was becoming an issue in customer satisfaction and sales cycles.

The prior search technology was Amazon Web Services (AWS) CloudSearch. Due to the rigid nature of AWS CloudSearch, its poor performance, and limited ability for easy and fast tuning for relevance, the system was not well-maintained, nor able to scale and deliver the experience that Pro customers were looking for. With "millions of searchable documents: 24 million votes, 700 000 amendments, 16,000 legislators, and 7 million searchable items" on the Pro platform, indexing of content and speed of answers were key requirements of any new search platform.

POLITICO's unique concept connecting different, but related information types from different geographies, ranging from complex legislative documents, to news and more, in a single platform created an interesting search challenge. Taking a look at the challenges facing POLITICO, it was clear that a robust, scalable solution was required: 

• Unique to the POLITICO Pro platform, the search functionality maintains a division between information types, displaying, but pre-filtering results in silos to clearly illustrate where results exist. Users are then able to further narrow down results with dynamic filters by information type. 
  • Challenge: singular topics have a lot of results produced across different information types, improvements to speed and the implementation of more advanced search operators is essential. 
• POLITICO Pro coverage includes legislative datasets from the European Union, United Kingdom, France, and Germany. Legislation includes many unique identifiers, including things like numbers, text, inter-institutional identifiers, transpositional identifiers, and more. Additionally, commonly referred to legislative editorial names are often not the official title (ex: Wifi4EU editorially is officially 2016/0287(COD) or Promotion of internet connectivity in local communities). 
  • Challenge: providing accurate results and high relevance, while accommodating for title discrepancies. 
• The policy making process is generally complex, so naturally, the legislative and legal texts scraped and indexed by POLITICO, supporting the work of users can include very specific and technical terminology. Not to mention, documents exceeding ten thousand characters is the norm. 
  • Challenge: ensuring large technical documents can be indexed and search keywords can be clearly highlighted for users, all while keeping a friendly developer experience top of mind.

The solution

After consideration and review of several different search options, POLITICO decided Algolia was the best fit. The out of the box functionality offering speed, relevance, and a top notch developer experience was perfect for their small (but growing) development team. 

Unique to the POLITICO Pro implementation, Algolia enabled the developer team to utilize a single index to accommodate multiple information types. In this context multiple information types include news, legislation, elected officials, court cases, press releases, transcripts, and more, all combined in their search results.

Technology spotlight: 

As a single piece of legislation can have multiple documents attached to it that we also want to match search results for, we added a secondary objectID field to our document called distinctObjectID. In this field, we then include the secondary objectID when it's for a document belonging to another type. For filtering on multiple types these objects will also be marked as belonging to both types, which we do by a simple boolean flag in a separate field we call flags. A stripped down version of a document belonging to a bill without the content would then look like this:

{

"objectID": "document_12389",

"distinctObjectID": "bill_90182",

"flags": ["is_type_document", "is_type_bill"]

}

By using distinct searches and disjunctive faceting, POLITICO Pro is able to provide counts on the number of different types matching a specific search in real time as you type and deliver an excellent user experience. 

“After considering several search solutions, Algolia’s speed, relevance, built in analytics, and developer experience stood out as the best for our needs. The implementation of Algolia represents a maturation of our Pro platform, a clear signal to users we’re investing in services that add immediate value to their work. Since launch, we’ve seen a sharp upward trend in usage and have involved the entire team to continuously optimize the user search experience through merchandising, rules, and reviewing analytics data.” Karl Laurentius Roos, Director Technology Development

Technology spotlight

Using the Algolia Search Helper, POLITICO built a custom front end solution for their search experience. The custom logic displays results before the content of the results are loaded. A core part of the POLITICO search experience is finding or sifting through legislation — as a user searches for legislation, they'll see results instantly while typing, seconds later seeing the full result. With quite heavy datasets, and wanting to avoid indexing all of this data, this custom implementation instantly calls their GraphQL API to fetch the missing pieces that are not indexed. Since the Pro platform performs the lookups by primary key, as the primary keys can be derived from the objectID of the indexed objects, this lookup is incredibly fast and can be run in parallel. For example, we get back a search result with the following objectID in hits: bill_91236, bill_3701, bill_65215. We would then generate a GraphQL query in our application layer and instantly call our API, looking like this:

query {

bill_91236 bill(id: 91236) {

id

...additional fields

}

bill_3701 bill(id: 3701) {

id

...additional fields

}

bill_65215 bill(id: 65215) {

id

...additional fields

}

}

All implementation and development was handled internally by POLITICO.

The result

On top of building an excellent search experience, with Algolia, POLITICO has developed a competitive advantage through search in a complex, competitive media landscape. POLITICO is the “go to” media source when it's last minute and users need to know something right now. With Algolia, great search now means users can be delighted with the search experience and will increasingly rely on POLITICO for their information gathering needs over the institutions from which POLITICO scrapes data.

With the technical infrastructure implemented, through the intuitive Algolia admin, POLITICO developers and non-technical members of their team are able to review and optimize search results, ensuring the user experience remains well curated. 

“The new search enables me to be more efficient in my day-to-day work, and has impacted my outreach in two ways. First, I can quickly conduct precise searches to find key mentions of my clients, their brands, or their topics of interest. Second, by setting up automated and complex keyword monitoring, I can share this information with clients and trigger interest in services previously not in their subscription. The overall value for me is realized through the speed and reliability (trust) of search.” Riccardo Dugulin, Account Executive POLITICO Pro 

Since implementing Algolia, POLITICO has seen the following benefits and measured results: 

• 8X faster search 
• Highly relevant results 
• Intuitive user experience 
• Not losing sales due to users not finding what they are looking for 

“This one’s exciting. We’re happy to release an all new search experience—this means it’s easier for you to find what you’re looking for through a new user interface and filters layout. It’s also fast, results are loaded almost instantly. And….(pause for effect) we support additional search operators like + and - to make queries more precise. Search operators can also be used to dial in your project monitors.” Internal POLITICO Newsletter

Looking ahead

Following their initial implementation, POLITICO has plans to further integrate Algolia capabilities into their search stack. A priority is expanding the role of Algolia to power both their Pro platform and free site, a significant upgrade to deliver a top notch discovery experience for all users. Additionally, things like trending searches, search suggestions, and autocomplete are details that will enable POLITICO to bring the most value to the user experience. 

Thinking bigger and longer term, POLITICO envisions a concept to further contextualize content discovery through newsletter “chunks.” Newsletters are the core of the Pro offering, providing the “need to know now” briefing in each user’s inbox each morning. The newsletters contain blurbs on key topics, people, companies, and more, by indexing all sections of newsletters in “chunks” to deliver the full picture driving the day ahead, or a single section or mention impacting a Pro client. 

With Algolia, POLITICO is able to quickly and continuously innovate, pushing the boundaries of a traditional news source to arm policy professionals with the insight they need to do more.