Building recommendations for ecommerce & media, with & without AI

As the popularity of ecommerce and digital media explodes, the tools most online companies use to make recommendations have not meaningfully changed – until recently. The early recommendation tools, around since the earliest digital marketplaces, based their recommendations on heuristic-based filters, breaking down items into categories and tracking co-occurrence in users’ carts or watch lists. 

While filtering-based recommendations (as discussed below) are simple to implement and don’t necessarily rely on complex algorithms, they have their limitations. Modern solutions based on machine learning offer a significant improvement.

Amazon is at the forefront of AI-powered recommender systems – recommendations account for 35% of their sales. Their algorithms are carefully guarded, very well-trained – and expensive, but it’s possible to get great results with much less investment. The same goes for Netlify and Spotify.

Recommendation engines can be built around a variety of models. Depending on your products and target consumers, different algorithms will be used according to your needs. Two of the most widely-used models will be very familiar to anyone who shops on Amazon. “Related Items” and “those-who-bought-this-also-bought” recommendations are both collaborative filtering algorithms, and there are a range of options for implementing them. 

In this post, we’ll cover a few different approaches to recommendations tools. You can use classic methods, based on tagging and filtering. Alternatively, we’ll show how machine learning (ML) and artificial intelligence (AI) can provide privacy-safe and more natural recommendations. Lastly, you’ll see how Algolia Recommend approaches these problems to provide developers an API-level recommendation engine. But first, it’s important to understand the user’s context and how they’ll see your recommendations.

Make high-quality recommendations part of the search journey

Potential customers generally enter a site with one of two approaches. They may know, with specific guidelines, exactly what they are looking for – a high-quality child’s bike helmet, a white blouse in natural fibers, a brown leather ottoman. Or they may be browsing generally, looking for a gift for a milestone birthday or better solutions for a home office. 

In either case, recommendations have the potential to connect potential buyers with products they might not have found on their own by identifying patterns in user behavior. Below, we’ll look at two possible approaches – a simple filter-based approach, and an ML-based solution.

Get custom results with simple tags with no AI

Among dozens of white blouses, a user clicks on one that is tunic-length linen. Without using any AI, you can make recommendations by using tags and sales history. 

Perhaps that blouse has been tagged as “casual” in style and is part of your Spring editorial campaign. You could easily return a short list of three or four other blouses that meet at least two criteria—casual style, spring season, linen, white, tunic-length, price point. 

Preparing your data to provide this type of recommendation requires some up-front work and ongoing validation to ensure all products in your inventory are indexed correctly. For each item, you must decide what properties you want to “match” on, and the relative importance of each matching property. When a new product is added to your catalog, it will need to be ‘tagged’ and assigned matching properties that might not be easy to upload automatically. A blouse’s color may populate automatically, but ‘style’ or ‘season’ may not. It’s possible to get precise, quality recommendations using this approach, particularly if your inventory does not change too often. 

Another non-AI approach is to use sales history. It’s a reasonable assumption that customers who have one purchase in common might have others in common, as well. Sales history could feed recommendations for other items purchased by people who bought the selected item. The query to find these product-buyer-product relationships will likely use more resources than the product-product relationship above, so it may be inefficient for large inventories. For companies with many categories of products, it may also result in some unwanted ‘noise’ in the recommendations (for example, that best-selling item shows up because it’s generally popular, not because it’s related to other products). In addition, when a new product is added to your catalog, there will be no purchase history to draw on. 

An additional drawback of the sales history approach is that it introduces consumer data directly into your matching algorithm. It’s certainly possible to do this in a fully secure and private way, but any time customer data is used in public-facing code, there is a greater risk of exposing private data. Avoiding data leaks is a commitment that takes time and knowledge. 

Depending on your situation, these downsides can be substantial. While good recommendations don’t require AI, you may be able to avoid handling user data and brute-forcing fine-tuned recommendations. Let’s look at what it looks like to dip into an ML-based approach.

Use ML-based AI solutions for better recommendations

The trouble with filter-based, non-AI recommendations is that they assume users know what they want and can describe it accurately. Sometimes that is the case. But often, the most effective recommendations are for items the user might not have sought out or uncovered on their own.

The major advantage of ML-based recommendations is to help with the problem of discovery. Shoppers for kids’ bike helmets likely have other needs in common—children’s safety equipment, and potentially outdoor activities, healthy eating, or alternative commutes, or other interests that are even more difficult to predict. 

Rather than trying to anticipate all of these connections, an AI algorithm notices patterns in shopper behavior that could be difficult to predict or describe. The recommendations an ML-based solution provides have an element of serendipity and ‘fit’ that helps them stand out. To begin observing these patterns, you start by collecting user events. However, as with personalization mentioned earlier, it’s important that you don’t expose private data.

Capture user events for high quality results with low privacy risk

The richest data available to ML-based recommendation engines is user events like clicks and conversions. Historically, recommendations have only reflected users’ current action – the most recent item clicked, the current search term. Search for a white blouse, and you will get white blouses. Tracking the sequence of user actions provides richer insight into a specific user’s goals and interests than a single event. A clickstream that goes from white blouses to pastel dresses and on to sandals provides much greater insight into a shopper’s interests than just the initial search term.

This sort of user data tracking can introduce unwanted privacy risks into your application. A fortunate upside of AI solutions is that they are a win for user privacy and data security, as well as providing better recommendations. They learn based on the past behavior of many, then make predictions according to a model.

ML-based solutions are tuned specifically to the way customers interact with your products and services during a given session. The specific search terms, clicks, and conversions from a user’s current visit to your site help you make recommendations tailored to what they are looking for at that moment. 

Collecting event data enables more than just recommendations. Tracking user view, click, and conversion events can tell you a great deal about how users are interacting with your site and products. You’ll want to categorize and store the event data, training your model with linear regression and other ML-based algorithms.

If you choose to use Algolia, its Insights API will be your hub for collecting and sending user event data to your Application. You can use that data for A/B testing and dynamic product re-ranking that responds to users’ choices, and to offer more tailored recommendations.

The code below shows a simple example of how data gets sent to Insights. An onClick event handler in your front end makes a simple POST request (shown here as you might emulate the event from cURL):

curl -X POST \
https://insights.algolia.io/1/events \
-H 'x-algolia-api-key: ${ADMIN_API_KEY}' \
-H 'x-algolia-application-id: ${APPLICATION_ID}' \
-H "Content-Type: application/json" \
-d '{
    "events": [
        {
          "eventType": "click",
          "eventName": "Product Clicked",
          "index": "products",
          "userToken": "user-123456",
          "timestamp": 1529055974,
          "objectIDs": ["9780545139700", "9780439784542"],
          "queryID": "43b15df305339e827f0ac0bdc5ebcaa7",
          "positions": [7, 6]
        },
      ]
    }'

A queryID ties the click event to a specific Algolia search, and is only valid within one hour of the search. A session-based token connects the click event to a user. There is little risk of exposing sensitive user data, because it simply isn’t collected. Meanwhile, the user can get recommendations that are responsive to their actions in the moment.

Use collaborative filtering to generate precise recommendations

The most common “frequently bought together” suggestions start with collaborative filtering algorithms. The basic goal for any recommendation AI is to find similarities between users and products, and make suggestions based on those similarities. Collaborative filtering algorithms are widely used in data science to make predictions about how a user will rate a product, based on how other users with similar preferences have rated that product. Many explanations of collaborative filtering use movies as an example. A simplified scenario:

• I love period dramas, and have given high ratings to TV shows like The Crown and Call The Midwife.
• I really dislike TV shows with too much fantasy, and have given low ratings to Sherlock and Dr. Who. My sister loves them!
• Pat and Sam also both love period dramas, and have rated The Crown very highly too.
• Pat also enjoys fantasy-based shows, and gave high ratings to those shows I disliked.
• Sam is more similar to me, so he rates fantasy TV shows lower.
• A new show is released that is categorized as mystery/crime, but which has both historical and fantasy elements. Pat rates it highly, Sam rates it lower.
• None of us have ever rated a mystery/crime show before this.
• Because my preferences match Sam’s in general, this show is not recommended for me. My sister’s preferences match Pat’s, and she gets the new show in her recommendation queue!

At scale, a collaborative filtering algorithm is more complex than this, though the basic principles are very similar. Scalability is a selling point for ML-based recommendation systems, unlike the non-AI solutions we discussed above. The larger the dataset, the more compelling the case is for using an AI algorithm, because there is more opportunity to find patterns in user behavior. Amazon relies heavily on collaborative filtering, leading to product recommendations that can feel serendipitous and uniquely personalized. 

Collaborative filtering algorithms fall into a few categories – user-based, item-based, hybrid, and matrix factorization. Even among hybrid systems, there are a range of options for how many approaches to combine and in what priority order. Each is best-suited for different types of data sets, depending on the number of users, type of items, density of ratings per user, and item categories. 

Use Algolia recommend to get an ML-based solution that’s always improving

While there is no single recommendation solution that makes sense for every purpose, in most cases, a very good option does not have to be a complicated one. For ecommerce companies, a collaborative filtering approach is still likely the best choice. 

Algolia Recommend delivers high-performance AI-powered recommendations in an API. The API form factor means it is fast to implement, particularly because of the simple front-end UI code snippets. It is also a way to ensure that your recommendations will be better over time. Those iterative improvements delivered through the API will automatically come through – no recoding on your part.

If you don’t have an Algolia Index, create an account and app. Implementing Algolia Recommend, standalone or with Algolia Search, will be an easy step towards getting the best results out of your ecommerce site. You can see it in action in two videos, or learn more from our in-depth guide. If you’re not familiar with Algolia, these resources can help deepen your understanding of Recommendation engines in general and give you a sense of how easy it is to work with Algolia.