SAN FRANCISCO, OCTOBER 2, 2024 – Algolia, the world’s only end-to-end AI Search platform, today announced the availability of its next-generation Crawler, a critical tool rebuilt to enable developers to ingest data into Algolia AI Search more quickly and easily while spending less time on configuration and maintenance. This automated web scraping program simplifies and accelerates the ingestion of data by scanning web pages of any size. 

A Powerful Tool for Seamless Data Ingestion
The next-generation Crawler is a powerful tool in data ingestion that effortlessly extracts content from an owned website. By navigating through linked pages, the Crawler quickly populates and updates Algolia AI Search indices when it detects changes automatically. The Crawler is tailored for developers seeking a fast and convenient method to implement search capabilities, freeing them up to work on their core product instead.

Tim Benniks, Developer Relations Leader at Outreach and Hygraph, said: “It’s really awesome that Algolia’s new Crawler parses schema.org formats on my website and recognizes that the content is actually a blog post. It just works!”

Streamlining Data Integration Across Multiple Sources
In scenarios where customers’ websites are managed by different teams or powered by various systems, the new Crawler shines as a highly convenient solution. It eliminates the traditionally time-consuming and resource-intensive processes associated with data integration, providing a swift time to value.

The Crawler can simplify data ingestion by sidestepping the complexities of building and maintaining an indexing pipeline. It is ideal in situations where the data structure is complex and there are limitations in the ability to send data to Algolia via API clients. 

Philip Schmitt, Software Consultant, TNG Technology Consulting noted that the Self-Service Crawler will help developers extract, format, and upload content from multiple sites to the Algolia AI Search platform: “The Algolia Crawler is a great fit for our corporate website. It aggregates distributed content rapidly, and automatically and periodically updates aggregated content without adding any complexity to our codebase. Moreover, it enables accurate and efficient search through records, thus enhancing the search experience for users.”

Additionally, Algolia provides data connectors so that users can connect to popular sources such as Google Analytics, Adobe Analytics, and CSV formats to further enrich their records and improve the relevance in their search results.

Introducing Usage-Based Pricing
The reimagined Algolia Crawler will be available to all current Algolia Price plans. This will include 10,000 free crawls per month as part of Algolia’s ‘start for free’ Build program for developers. Developers can start experimenting with the new Crawler immediately and at no cost. 

Bharat Guruprakash, chief product officer, said: “The Algolia next-generation Crawler is available to developers, business users, and enterprise customers, and it will be integrated into the AI Search onboarding process. This easy-to-use and fast, low code solution enables data ingestion in minutes and is particularly beneficial for users who wish to build a proof of concept and experiment with it quickly, without initially uploading sensitive data.”

Getting started in minutes with No/Low code Configuration
The new Crawler features guided onboarding with intuitive step-by-step directions to facilitate a fast and easy setup. This streamlined approach includes a visual alternative to a traditional code editor – a No-Code Configurator for common settings (e.g., scheduling specific times for automatic crawls, listing of pages to be ignored, inputting login credentials for password-protected sites, and more).

Algolia continues to innovate and provides tools, such as the Self-Service Crawler, to streamline complex processes, enabling developers to focus on their core project. Developers can try the Self-Service Crawler and AI Search for free here.

About Algolia
Algolia is a leading provider of AI search solutions, serving over 17,000 businesses and 500,000 developers globally. Renowned for its user-friendly API-First platform and the fastest AI search technology, Algolia is the largest hosted search engine, trusted by businesses and developers for 1.75 trillion searches per year. Backed by a decade of innovation, expertise, and growth, Algolia continuously redefines the search landscape with its commitment to user-friendly solutions, significant scalability, and unmatched speed. Learn more: www.algolia.com.