Search relevance is the measure of accuracy of the relationship between the search query and the search results.
Today’s online users have high expectations. Thanks to the high bar set by sites like Google, Amazon, and Netflix, they expect accurate, relevant, and rapid results. However, the reality is many sites do not have optimized results pages that understand the user’s intent and bring them to their needs with ease.
If you’ve ever searched a website only to be shown a bunch of useless, unrelated results, then you know just how your users might feel: frustrated and primed to go to a competitor’s site to find results. Search relevance is integral to the user experience.
Website owners can fine-tune their search relevance to order search results in the most helpful way to users. This can be based on a number of factors, such as search intent, business priorities, textual relevance, spelling accuracy, geolocation of the user, or proximity of the keywords in the content searched.
Fine-tuning search relevance for accuracy
Relevance can be hard to get right, since it is highly dependent on context and a number of changing variables. For example, the type of site matters: the way things should be ranked on an e-commerce website versus an academic site won’t be the same. The type of searcher matters as well. A result that is relevant for a customer might not be relevant for the business owning the search engine, and vice versa. Additionally, different people will have different ways to express what they are looking for, and even for the exact same query, different users will expect different results. A result ranking formula has to work with these nuances.
Why does search relevance matter?
Optimizing search relevance is an extremely important, yet often overlooked, facet of user experience design. Research shows 43% of website visitors go immediately to the search bar, and these searchers are about 2-3 times more likely to convert. When users are served results that align to their query and interests, they will be more satisfied, more engaged, and even more likely to convert.
In addition, modern online users have high expectations for website usability and thus ease of use and simplicity of design are important factors in how customers perceive a brand.
A brief history of search relevance
The history of search relevance goes back to the early days of the Internet, when researchers were trying to figure out methods for information retrieval as well as how to explore all the new content that was being created. This quickly led to the invention of the search engine.
Early search engines and protocols such as Archie, created in 1990 by a postgraduate student at McGill University, and Gopher, created in 1991 by researchers at the University of Minnesota were important milestones in the development of modern search relevance systems. They enabled researchers to use search terms to search through the file systems of other institutions that they were connected to over the Internet.
Yet, they were still very technical systems that required users to have advanced knowledge of computers and low-level Internet concepts. However, just a couple years later in 1993, the World Wide Web started to flourish as hundreds of websites started to go online, initiating a whole new wave of search systems.
Early web search engines
Soon web crawling was invented to automatically load and update web pages into the search engine indexes, which allowed for much more content to be searched.
Web search engines such as Excite in 1993 and Yahoo in 1994 quickly gained popularity due to their ease of use. They even included some basic statistical models aimed at understanding user queries and how they related to content.
These novel early systems employed a workable but limited method to order the most relevant search results to users. That is, a lot of relevancy ranking was based on the number of times keywords appeared in web pages and didn’t consider any other criteria to assess the quality of the web pages.
Then Google came onto the scene. Google, founded September 4, 1998 in Menlo Park, CA, vastly improved search relevance and the search box, by building cutting-edge search engine technology.
Throughout the 2000s, for instance, search engines began building more statistical systems for interpreting query semantics, predicting relationships between different keywords, and using click-through data to dynamically adjust results. As search engine optimization (SEO) professionals began to learn how these algorithms worked, search engines also had to keep up and defend themselves against more sophisticated attempts to “game” the system, so that the results stayed as fair as possible.
Creating relevant document search
With the number of sites expanding, the need to search for relevant documents within particular sites and databases grew as well. Document search was an important precursor to the way we search on sites today.
Traditional ranking systems would often look at the frequency of keywords in documents to predict their relevance. For instance, a classic algorithm known as TF-IDF would look at the number of times keywords appeared in the respective documents (Term Frequency) and at the number of times keywords appeared in all other documents in the repository (Inverse Document Frequency). The latter analysis helps to filter out common words that are typically noise, such as prepositions.
While these early approaches to relevancy such as TF-IDF were good at general purpose document search, they failed to take advantage of the additional structure and metadata that is afforded in most websites. Modern content has titles, descriptions, categories, tags, and more that can be used to interpret site content and improve search relevance.
Today’s search relevance
Today, as websites have grown their content and product offerings, optimizing search relevance is a major consideration for individual site search engines. Businesses operating their own relevancy systems need to take into account their specific business needs to make their search useful.
For instance, an e-commerce brand may have thousands of diverse products and customers of very different demographics. Thus, when a customer searches for a product, the internal search engine must be able to provide results that are not only related to the query, but also contextually relevant to the specific user.
Additionally, marketers may want to promote seasonal items similar to that of their in-store merchandising efforts, or business operators may want to push higher margin goods. Therefore, a relevant search system must also be able to take these factors into account and provide custom ranking that can be adjusted to meet these needs over time.
Many of these algorithms are still clunky, however. Some algorithms, though, like Algolia’s, take into account factors such as the importance of the matching attribute and the proximity of the keywords. By doing so, the search results are much more likely to be relevant to users than general purpose search algorithms.
More recently, to improve relevance, search designers have been working to build in more personalization and contextualization. This includes things like machine learning and natural language processing to enable more conversational search, tracking of user search and browsing history to allow for custom interpretations of queries, and automatic tagging and categorization of web pages for an understanding of content at a higher level than simply that of text keywords.
Fine-tune your search relevance with Algolia
Optimizing a website’s search relevancy is a complex and ongoing process. It requires not only providing results that match users’ queries, but also providing personalized results and meeting specific business needs. Further, as users move more toward voice-enabled devices and digital assistants, businesses will have to figure out how to provide a new type of interface that can speak naturally with users.
To provide all of these features to your customers, you’ll need a search as a service partner that provides all the industry best practices out of the box. Learn how Algolia can help provide a personalized and relevant search experience for your users.