Search by Algolia
Vector vs Keyword Search: Why You Should Care
ai

Vector vs Keyword Search: Why You Should Care

Search has been around for a while, to the point that it is now considered a standard requirement in many ...

Nicolas Fiorini

Senior Machine Learning Engineer

What is AI-powered site search?
ai

What is AI-powered site search?

With the advent of artificial intelligence (AI) technologies enabling services such as Alexa, Google search, and self-driving cars, the ...

John Stewart

VP Corporate Marketing

What is a B2B marketplace?
e-commerce

What is a B2B marketplace?

It’s no secret that B2B (business-to-business) transactions have largely migrated online. According to Gartner, by 2025, 80 ...

Vincent Caruana

Sr. SEO Web Digital Marketing Manager

3 strategies for B2B ecommerce growth: key takeaways from B2B Online - Chicago
e-commerce

3 strategies for B2B ecommerce growth: key takeaways from B2B Online - Chicago

Twice a year, B2B Online brings together industry leaders to discuss the trends affecting the B2B ecommerce industry. At the ...

Elena Moravec

Director of Product Marketing & Strategy

Deconstructing smart digital merchandising
e-commerce

Deconstructing smart digital merchandising

This is Part 2 of a series that dives into the transformational journey made by digital merchandising to drive positive ...

Benoit Reulier
Reshma Iyer

Benoit Reulier &

Reshma Iyer

The death of traditional shopping: How AI-powered conversational commerce changes everything
ai

The death of traditional shopping: How AI-powered conversational commerce changes everything

Get ready for the ride: online shopping is about to be completely upended by AI. Over the past few years ...

Aayush Iyer

Director, User Experience & UI Platform

What is B2C ecommerce? Models, examples, and definitions
e-commerce

What is B2C ecommerce? Models, examples, and definitions

Remember life before online shopping? When you had to actually leave the house for a brick-and-mortar store to ...

Catherine Dee

Search and Discovery writer

What are marketplace platforms and software? Why are they important?
e-commerce

What are marketplace platforms and software? Why are they important?

If you imagine pushing a virtual shopping cart down the aisles of an online store, or browsing items in an ...

Vincent Caruana

Sr. SEO Web Digital Marketing Manager

What is an online marketplace?
e-commerce

What is an online marketplace?

Remember the world before the convenience of online commerce? Before the pandemic, before the proliferation of ecommerce sites, when the ...

Catherine Dee

Search and Discovery writer

10 ways AI is transforming ecommerce
e-commerce

10 ways AI is transforming ecommerce

Artificial intelligence (AI) is no longer just the stuff of scary futuristic movies; it’s recently burst into the headlines ...

Catherine Dee

Search and Discovery writer

AI as a Service (AIaaS) in the era of "buy not build"
ai

AI as a Service (AIaaS) in the era of "buy not build"

Imagine you are the CTO of a company that has just undergone a massive decade long digital transformation. You’ve ...

Sean Mullaney

CTO @Algolia

By the numbers: the ROI of keyword and AI site search for digital commerce
product

By the numbers: the ROI of keyword and AI site search for digital commerce

Did you know that the tiny search bar at the top of many ecommerce sites can offer an outsized return ...

Jon Silvers

Director, Digital Marketing

Using pre-trained AI algorithms to solve the cold start problem
ai

Using pre-trained AI algorithms to solve the cold start problem

Artificial intelligence (AI) has quickly moved from hot topic to everyday life. Now, ecommerce businesses are beginning to clearly see ...

Etienne Martin

VP of Product

Introducing Algolia NeuralSearch
product

Introducing Algolia NeuralSearch

We couldn’t be more excited to announce the availability of our breakthrough product, Algolia NeuralSearch. The world has stepped ...

Bernadette Nixon

Chief Executive Officer and Board Member at Algolia

AI is eating ecommerce
ai

AI is eating ecommerce

The ecommerce industry has experienced steady and reliable growth over the last 20 years (albeit interrupted briefly by a global ...

Sean Mullaney

CTO @Algolia

Semantic textual similarity: a game changer for search results and recommendations
product

Semantic textual similarity: a game changer for search results and recommendations

As an ecommerce professional, you know the importance of providing a five-star search experience on your site or in ...

Vincent Caruana

Sr. SEO Web Digital Marketing Manager

What is hashing and how does it improve website and app search?
ai

What is hashing and how does it improve website and app search?

Hashing.   Yep, you read that right.   Not hashtags. Not golden, crisp-on-the-outside, melty-on-the-inside hash browns ...

Catherine Dee

Search and Discovery writer

Conference Recap: ECIR23 Take-aways
engineering

Conference Recap: ECIR23 Take-aways

We’re just back from ECIR23, the leading European conference around Information Retrieval systems, which ran its 45th edition in ...

Paul-Louis Nech

Senior ML Engineer

Looking for something?

facebookfacebooklinkedinlinkedintwittertwittermailmail

Language is one of our most basic ways of communicating, but it is also a rich source of information and one that we use all the time, including online. What if we could use that language, both written and spoken, in an automated way? That’s what natural language processing sets out to do.

Natural language processing, or NLP, takes language and processes it into bits of information that software can use. With this information, the software can then do myriad other tasks, which we’ll also examine.

Why is natural language processing necessary?

But first, why is natural language processing even necessary? First off is the fact that massive amounts of information is created and shared every day through natural language. Billions of social media posts come through daily. Trillions of searches happen on search engines great and small. Call transcripts. Emails. Classifieds. News articles. Some of these, like search queries, benefit directly from NLP. Others, like news articles, can be processed via NLP to create value.

Let’s look at a couple of examples in more detail. We’ll start off by looking at news articles.

Jones to Assume Presidency of Acme Corp.

Marcus L. Jones today announced that he was to become the 4th President in Acme Corp. history. He will lead the widget maker into its next chapter as it examines expansion into new markets, such as Europe, Mexico, and Canada.

Now think about all of the things we may want to do with this text. For example, we could want to know which companies, subjects, countries, and other key entities are mentioned so that we can tag and categorize similar articles. One way we can do that is to first decide that only nouns and adjectives are eligible to be considered for tags. For this we would use a parts of speech tagger that will specify what part of speech each word in a text is.

NLP normalization and tokenization

But even once we identify those words, things are tricky, because is “widget” different from “widgets”? Of course not! So we need to use some normalization, which will collapse words to their core so that different variations can be considered equivalent. And normalization can be complex, in cases such as “Europe” and “EU,” or “Marcus L. Jones” and “Marcus Jones.”

This approach even ignores that the items relevant for tagging and categorization may not be single words, but could even be a phrase, such as “Acme Corp.” Identifying these items is the job of tokenization. Tokenization breaks a larger text into smaller pieces. It may break a document into paragraphs, paragraphs into sentences, and sentences into “tokens.” (We won’t say words here, because Acme Corp. can be a token but isn’t a word, and “isn’t” is a word, but would often be broken down into two tokens: is and n’t.) Tokenization can be very difficult. For example, even something as “simple” as identifying sentences in a paragraph is tricky, because what happens when you have a sentence like the first one in the article? Is “Marcus L.” a sentence because it ends with a paragraph and is followed by a word with a capital letter?)

Using NLP for named entity recognition

Altogether, identifying key concepts is what is known as named entity recognition. Named entity recognition is not just about identifying nouns or adjectives, but about identifying important items within a text. In this news article lede, we can be sure that Marcus L. Jones, Acme Corp., Europe, Mexico, and Canada are all named entities.

Finally, we may want to understand the connections between words. This will help our programs understand the semantics behind who the “he” is in the second sentence, or that “widget maker” is describing Acme Corp.

Natural language processing for search

Natural language processing for search queries is just as important, but with its own challenges and needs. A good way to illustrate this is to also discuss an important factor of natural language processing: the fact that there are thousands of natural languages spoken in the world. Different languages will have different needs, and while English is the language that much NLP software starts with, it is not indicative of all languages.

As an example, English rarely compounds words together without some separator, be it a space or punctuation. In fact, it is so rare that we have the word portmanteau to describe it. Other languages do not follow this convention, and words will butt up against each other to form a new word entirely. In German, the word “Hundehütte” means dog house. It’s not two words, but one, but it refers to these two concepts in a combined way.

A naive search engine will match Hundehütte to Hundehütte well enough, but it won’t match that query word to the phrase “Hütte für große Hunde,” which means house for big dog. Natural language processing comes in to decompound the query word into its individual pieces so that the searcher can see the right products. This illustrates another area where  the deep learning element of NLP is useful, and how NLP often needs to be language-specific.

Natural language processing structures data for programs

Going through all of these steps for natural language processing (altogether known as the natural language processing pipeline) returns information that is structured in a way that software can understand. By now you’ve seen that there is a lot of information hidden inside language. A 40 word paragraph can refer to a company, a person, three regions, and much information about those items. Humans are very good at identifying the important parts of language, and understanding how it goes together, but bad at taking hundreds, thousands, or millions of texts and finding trends or grouping them together. Most software programs are the reverse: they can find trends or categorize texts, but they are bad at the text itself. That’s why we use tailored software, natural language processing, to structure the text into a way that those programs can use. (By the way, why not combine all the steps into a single program? Having small, focused programs can make each step better, and allow us to combine different tools for different purposes.)

Conclusion

Above, we looked at the examples of a news article and a search query, and how we could use natural language processing to better transform the text. Think now about the other examples of textual content we discussed, like call transcripts, classifieds, or emails. What kind of processing might these texts need?For more information on how Algolia’s search and discovery APIs leverage NLP, or to learn more about how we can help you implement this powerful technology within your site or app for a more engaging user experience, please contact our team of experts.

About the author
Dustin Coates

Product and GTM Manager

linkedin

Recommended Articles

Powered byAlgolia Algolia Recommend

Advanced keyword search is built upon natural language processing (NLP)
ai

Julien Lemoine

Co-founder & former CTO at Algolia

NLP & NLU as part of semantic search
ux

Dustin Coates

Product and GTM Manager

What is natural language search?
product

Dustin Coates

Product and GTM Manager