Learn how a website crawler can help you increase your digital revenue, and check out Algolia’s hosted crawler for businesses.

Search & Discovery Writer

Engineering

Read articles about search, SaaS, APIs, infrastructure and learn about new Algolia features.

Algolia Blog

https://blog-api.algolia.com/wp-content/uploads/2021/08/webcrawler_3-03-min-scaled.jpg

What is a web crawler? - Algolia Blog

Home

What is a web crawler?

https://secure.gravatar.com/avatar/c5079cf534cf5ed1038ab51f2ea24302?s=96&d=mm&r=g

Catherine Dee

Head of Algolia R&D Romania | AI Evangelist

In this fourth post, we’ll be building a recommendations model - content-based and collaborative filtering - for a recommender engine.

https://blog-api.algolia.com/wp-content/uploads/2021/07/BP-illu-steps-3-1.png

The anatomy of high-performance recommender systems – Part IV - Algolia Blog

The anatomy of high-performance recommender systems – Part IV

Head of Algolia R&amp;D Romania | AI Evangelist

https://secure.gravatar.com/avatar/f980a1b7c39927e74d901ab821f04f69?s=96&d=mm&r=g

Ciprian Borodescu

Senior Developer Relations Specialist

Setting up an Omnibar Autocomplete solution to fire interactions with JavaScript.

https://blog-api.algolia.com/wp-content/uploads/2017/10/2017-10_Multi-Cluster.png

Creating an omnibar with Autocomplete - Algolia Blog

Creating an omnibar with Autocomplete

https://secure.gravatar.com/avatar/aed4e7d7aacb0a1b09c1b45bf4510b4d?s=96&d=mm&r=g

Bryan Robinson

Check out the live demo here: http://alg.li/NYiGIR

Building a search UX in 45 seconds with Algol…

Building a search UX in 45 seconds with Algolia

The topic of this new Live Coding Session is "Autocomplete Omnibar - Trigger user interactions directly from your search box with Autocomplete.js".

Bryan Robinson, Sr Developer Advocate at Algolia, will walk you through setting up Autocomplete to fire interactions with JavaScript. Specifically, we'll build an omnibar to toggle light and dark mode for our website. An omnibar is a search field that has both search and actions that can be taken. A strong example of this is the Chrome or Firefox search and URL bar.

In the search field, a user will be able to type in / commands. These commands will be tied to specific JavaScript methods to trigger. We'll also make the Autocomplete results stateful. When the app is in light mode, the light mode option will show an "enabled" flag. When dark mode is enabled, the dark mode option will show the flag.

Trigger user interactions directly from your …

Trigger user interactions directly from your search box with Autocomplete.js

After search and navigation, Algolia is now adding a recommendations API to the developer’s toolkit, giving you the ability to optimize across the entire product catalog.

In this Live Coding Session, we will show you:

- The 4-step process to add recommendations to your site
- The fully packaged "frequently bought together" and "related products" widgets
- How to completely customize your recommendations
- All the filters to refine the carousel purpose (items coming from the same category, same colour, in a specific range of prices, etc.)
- The fallback strategy when there’s not enough data for our models to generate recommendations
- How Algolia Recommend can be leveraged on all your websites, from the product detail page to the checkout.

Try out Algolia on www.algolia.com!

Adding product recommendations to your websit…

Adding product recommendations to your website in 6 lines of code

- Part 1: Introduction & Data indexing - https://www.youtube.com/watch?v=KRzrHu_XVVY&ab_channel=Algolia
- Part 2: Geo-search implementation using Algolia and Mapbox: https://www.youtube.com/watch?v=jyntg6hkKMs&ab_channel=Algolia 
- Part 3: Mapbox address search in an Autocomplete & Click & Collect logic implementation using Twilio 

In this series of Live Coding Sessions, we show how to build a store locator in React using Algolia, Mapbox and Twilio.

E-commerce shoppers expect convenience and want the physical and online worlds to mesh and allow them to conduct their business on whichever channel they want. For example, users may choose to:
- Search products online, then purchase in store
- Browse and compare products in person, then purchase online
- Shop and purchase products online, then pick up the purchase in person
- Return previously purchased items in person, rather than shipping them back

At the end of these sessions, we’re hoping you’ll have the main keys to building a Store locator your user can use to easily find the Nearest Store, or to “Buy Online, Pick up In-Store” (BOPIS) / “Click & Collect” products they’d like to get in no time.

Try out Algolia on www.algolia.com

Building a store locator in React using Algol…

Building a store locator in React using Algolia, Mapbox and Twilio - Part 3

- Part 1: Introduction & Data indexing - https://www.youtube.com/watch?v=KRzrHu_XVVY&ab_channel=Algolia
- Part 2: Geo-search implementation using Algolia and Mapbox
- Part 3: Click & Collect logic implementation using Twilio - https://www.youtube.com/watch?v=rKFMPYDwOn4&ab_channel=Algolia

In this series of Live Coding Sessions, we show how to build a store locator in React using Algolia, Mapbox and Twilio.

E-commerce shoppers expect convenience and want the physical and online worlds to mesh and allow them to conduct their business on whichever channel they want. For example, users may choose to:
- Search products online, then purchase in store
- Browse and compare products in person, then purchase online
- Shop and purchase products online, then pick up the purchase in person
- Return previously purchased items in person, rather than shipping them back

At the end of these sessions, we’re hoping you’ll have the main keys to building a Store locator your user can use to easily find the Nearest Store, or to “Buy Online, Pick up In-Store” (BOPIS) / “Click & Collect” products they’d like to get in no time.

Try out Algolia on www.algolia.com

Building a store locator in React using Algolia, Mapbox and Twilio - Part 2

- Part 1: Introduction & Data indexing - Blogpost at https://www.algolia.com/blog/engineering/building-a-store-locator-in-react-using-algolia-mapbox-and-twilio-part-1/
- Part 2: Geo-search implementation using Algolia and Mapbox: https://www.youtube.com/watch?v=jyntg6hkKMs
- Part 3: Click & Collect logic implementation using Twilio - https://www.youtube.com/watch?v=rKFMPYDwOn4

In this series of Live Coding Sessions, we show how to build a store locator in React using Algolia, Mapbox and Twilio.

E-commerce shoppers expect convenience and want the physical and online worlds to mesh and allow them to conduct their business on whichever channel they want. For example, users may choose to:
- Search products online, then purchase in store
- Browse and compare products in person, then purchase online
- Shop and purchase products online, then pick up the purchase in person
- Return previously purchased items in person, rather than shipping them back

At the end of these sessions, we’re hoping you’ll have the main keys to building a Store locator your user can use to easily find the Nearest Store, or to “Buy Online, Pick up In-Store” (BOPIS) / “Click & Collect” products they’d like to get in no time.

Try out Algolia on www.algolia.com

Building a store locator in React using Algolia, Mapbox and Twilio - Part 1

In this video, you will learn how to use the Algolia Firestore Extension to sync the documents between a Firestore collection and an Algolia Index automatically. You will also learn how to use the Algolia JS SDK to implement blazingly fast search in an Ionic 5 app using just 5 lines of code.

#algolia #firestore #firebase #samarth #samarthagarwal #extension #cloud #flutter #ionic #ionicframework #ionicalgolia #search #fulltextsearch #searchfirestore #firestoretextsearch #firestorealgolia

Follow me on Twitter @Samarth_Agarwal to learn about all the crazy stuff I do with Flutter and other technologies. Visit samarthagarwal.in for getting in touch regarding project development and consultation requirements.

Algolia Cloud Firestore Extension with Ionic …

Algolia Cloud Firestore Extension with Ionic 5 App

Build beautiful, fast, and relevant search for Firebase apps.

Full-text search of your Cloud Firestore data, now in minutes with the Firebase Algolia extension.

Light a fire on your search: https://www.algolia.com/developers/firebase-search-extension/

Firestore search with Algolia

Implementing search on an ecommerce site can be trickier than it sounds. Between knowing what to sync, how to combine data and achieving the most efficient front-end interface for your users, it can take a while before achieving a great website.

In 30 minutes, Guy Marriott and Matthew Foyle will:

- Take an existing Commerce.js storefront
- Set up a data pipeline so that the product catalogue is synchronized with Algolia in real time using serverless functions
- Set up best in retail search configuration
- And finally, integrate into the existing frontend, in order to achieve a great commerce experience for an end user

Live Coding Session - Integrate search into y…

Live Coding Session - Integrate search into your Commerce.js store in 30 min

This video was recorded on February 4th, 2021 during our Algolia Community Party - Online edition.

Reddinx: Search your favorite subreddit with Algolia - Tim Carry, Full Stack Engineer at Doctolib

As a Dungeon and Dragons player, I love the /dndmaps subreddit. But it's tough to find that great map I saw 2 months ago. So I built `reddinx`, a tool to crawl data from any subreddit and push it toAlgolia. I then added an infinite-search on top of it and automated the process so any new map is automatically added to the index. And the best part? It's open source. Come to the talk to learn how it works, and use it for your own favorite subreddit.

Reddinx: Search your favorite subreddit with …

Reddinx: Search your favorite subreddit with Algolia - Tim Carry, Doctolib

This video was recorded on February 4th, 2021 during our Algolia Community Party - Online edition.

Adding search to a Github social network using Algolia and NuxtJS - Tom Klotz, Software Engineer at Algolia

Openetwork is a social network. Its goal is to promote open-source project and create better interactions between people in the open-source community. This talk will cover how we implemented Algolia in our back-end in Node and our front-end in VueJs. We will try to provide tips for you to avoid falling in our mistake.

Adding search to a Github social network usin…

Adding search to a Github social network using Algolia and NuxtJS - Tom Klotz, Algolia

This video was recorded on February 4th, 2021 during our Algolia Community Party - Online edition.

A (cocktail) recipe for success - Mark Biek, Developer Lead at VIA Studio

From 1935-2012, the Mr. Boston Bartender's Guide was printed more than 11 million times and holds a place of honor on innumerable bars throughout the world. We decided to take it online, and used Algolia to create a blazing fast search experience. We'll go through the behind-the-scenes of the project in this talk.

A (cocktail) recipe for success - Mark Biek, …

A (cocktail) recipe for success - Mark Biek, VIA Studio

In this talk, we're going to build a SaaS app where people can publish a blog. 
We're going to ensure every blog use their dedicated fully-qualified domain and all data are safely separated while keeping everything in a single database.

Building multi-domain multi-tenant SaaS app w…

Building multi-domain multi-tenant SaaS app with JetStream - Julien Bourdeau, Algolia

Haroen Viaene (https://twitter.com/haroenv) presents how to plug the Algolia search service in a Gatsby-powered website — and introduces the Open Source Gatsby plugin he authored, gatsby-plugin-algolia (https://www.gatsbyjs.org/packages/gatsby-plugin-algolia/)

Haroen Viaene (https://twitter.com/haroenv) présente comment brancher le service de Seach-as-a-Service d'Algolia dans un site Gatsby — et il présente aussi son plugin Open Source, gatsby-plugin-algolia (https://www.gatsbyjs.org/packages/gatsby-plugin-algolia/).

Haroen Viaene — Search for a Gatsby site

Recorded during Algolia Search Party - Kubernetes edition - March 2019, Paris.

How to kill the Algolia dashboard during Black Friday or How not to use Kubernetes - Rémy-Christophe Schermesser, Algolia

Let’s talk about something that didn’t work out as expected. How the architecture of a system resulted in a cascading failure caused by kubernetes bad usage.

How to kill the Algolia dashboard during Blac…

How to kill the Algolia dashboard during Black Friday - Rémy-Christophe Schermesser

Recorded during Algolia Search Party - Happy Holidays to the community - December 2018, Paris.

Another take on Spotify search - Alexandre Stanislawski

Another take on Spotify search - Alexandre St…

Another take on Spotify search - Alexandre Stanislawski

Filmed during Vue.js Paris #19 at Algolia's office, January 2019

Abstract: The Algolia Docs Squad at Algolia just redesigned their entire docs and added a Vue-powered interactive showcase for InstantSearch, their front-end search library. In this talk, we dive into the architecture and technical choices of the project, how we used scoped slots to customize the rendering of every single detail, and how Vue InstantSearch lets you create new widgets with the power of mixins.

Twitter : https://twitter.com/frontstuff_io
Github : https://github.com/sarahdayan

Building an interactive showcase with Vue.js …

Building an interactive showcase with Vue.js & Vue InstantSearch - Sarah Dayan, Algolia

Jamstack Lightning Launch: Algolia Search Cra…

Jamstack Lightning Launch: Algolia Search Crawlers

Recorded during Algolia Search Party - Crawling edition - June 2019, Paris

The challenges of crawling the web — and how to overcome them - Samuel Bodin, Algolia

The world of crawling is dark and full of errors. I will talk about edge cases and present solutions we found at Algolia to overcome them.

The challenges of crawling the web — and how …

The challenges of crawling the web — and how to overcome them - Samuel Bodin, Algolia

Recorded during Algolia Search Party - Crawling edition - June 2019, Paris

Writing a distributed crawler architecture - Nenad Tičarić, TNT Studio

While at first glance writing a crawler might seem like a simple traversal of the web graph, in praxis it gets complicated by the many demands of the system. It has to be distributed, scalable, polite, robust and extensible but still performant. This talk will cover a PHP based solution to some of those problems.

Writing a distributed crawler architecture - …

Writing a distributed crawler architecture - Nenad Tičarić, TNT Studio

Recorded during Algolia Holiday Search Party - December 2019, Paris.

Hacking image search with Algolia - Maël Galite, Solutions Engineer at Algolia

Hacking image search with Algolia - Maël Gali…

Hacking image search with Algolia - Maël Galite, Algolia

Recorded during Algolia Holiday Search Party - December 2019, Paris.

Simplifying parcel delivery at Algolia - Clément Denoix, Software Engineer at Algolia

Simplifying parcel delivery at Algolia - Clém…

Simplifying parcel delivery at Algolia - Clément Denoix, Algolia

Recorded during TechLunch - Mobile Experience - November 2019, Algolia office, Paris

Robert Mogos, Engineering Manager - InstantSearch Web & Mobile, Algolia

Robert has been working for Amazon and Apple before joining Algolia. For the last 2 years he is focusing on building search and discovery experiences for both mobile and web. During the TechLunch he is talking about the lessons learned building mobile experiences.

Lessons learned building mobile experiences -…

Lessons learned building mobile experiences - Robert Mogos, Algolia

This video was recorded during our TechLunch #32 - Engineering Management on January 22nd, 2020 at Algolia's Paris office.

• Melinda Seckington, Technical Manager, FutureLearn

"Level Up: Developing Developers"

Melinda is a Technical Manager at FutureLearn, a social learning platform. She loves attending BarCamps, Hackdays and other tech meetups, and since 2009 has been organising them at Geeks of London. She also writes at MissGeeky, a blog about all things geeky and girly. This time Melinda will talk about how you can help level up your developers by using competencies and career development frameworks, what types of processes you can introduce to support personal development and how developers should be using opportunities like these to get a better understanding of what skills they should focus on.

Level Up: Developing Developers - Melinda Sec…

Level Up: Developing Developers - Melinda Seckington, FutureLearn

This video was recorded on April 28th, 2020 during our Algolia Community Party - Online edition #1.

Fury road to a highly available worldwide API - Matthew Foyle, Sr. Solutions Engineer at Algolia

Learn how Algolia reached high availability with bare metal servers and a distributed architecture

Fury road to a highly available worldwide API…

Fury road to a highly available worldwide API - Matthew Foyle, Algolia

Developers, we're bringing you a new format to learn and discover from your home: the Live Coding Sessions! We'll bring you each time some Algolia developers who will code something fun using the Algolia API + another tool, or building something creative with it.

For this first session, the topic is Algolia + commercetools. Join Matthew Foyle to learn in real time how to integrate headless commerce with best in class search and discovery.

Live Coding Session - Algolia + commercetools

This video was recorded on June 11th, 2020 during our Algolia Community Party - Online edition #2.

DocSearch: Enhancing the developer experience of open-source documentation - Sylvain Pace, Full-Stack Software Engineer at Algolia

In this talk, we will show you what is the state of the DocSearch project, which started by being a Holiday gift and now reaches more than 3.4 millions users a month. We will present you the new version of our UI. Also, we will see what was the process to make it happen and how it has been done technically. Join us to see how Algolia gives back to the community !

DocSearch: Enhancing the developer experience…

DocSearch: Enhancing the developer experience of open-source documentation - Sylvain Pace, Algolia

Rules can be useful to transform the search query itself, like detecting and automatically applying filters.

Intent Detection with Rules

Title: Advanced Search with Laravel and Algolia

If you're interested in having a highly relevant search experience on your website, you're in luck. In this Livestream, Nuno Maduro, an employee of Algolia, will show you how to build a search experience that your users will love.

Together, we'll build a Laravel application from scratch, and use Laravel Scout as a starting point to cover all necessary installation steps, from indexing your models to searching.

Then, you'll get a taste of Algolia's unique features with its Scout Extended, such as zero-downtime reimports and site-wide search among multiple models with Aggregators.
Finally, we are going to drive the brand new InstantSearch, building a find-as-you-type blazing-fast search experience. Sounds cool? Let's get started.

Algolia: https://www.algolia.com/
Laravel: https://laravel.com/

Music used during the livestream:
––––––––––––––––––––––––––––––
chill. by sakura Hz https://soundcloud.com/sakurahertz
Creative Commons — Attribution 3.0 Unported  — CC BY 3.0 
Free Download / Stream: http://bit.ly/chill-sakuraHz
Music promoted by Audio Library https://youtu.be/pF2tXC1pXNo
––––––––––––––––––––––––––––––

Algolia developer hub

Community, documentation, integrations and events.

Documentation

Community forum

Integrations

Algolia for Open Source

Engineering blog posts

What is a web crawler?

The anatomy of high-performance recommender systems – Part IV

Creating an omnibar with Autocomplete

Latest videos

Upcoming events

Free, out-of-the box search for your documentation