Search by Algolia
Easily integrate Algolia into native apps with FlutterFlow
engineering

Easily integrate Algolia into native apps with FlutterFlow

Algolia's advanced search capabilities pair seamlessly with iOS or Android Apps when using FlutterFlow. App development and search design ...

Chuck Meyer

Sr. Developer Relations Engineer

Algolia's search propels 1,000s of retailers to Black Friday success
e-commerce

Algolia's search propels 1,000s of retailers to Black Friday success

In the midst of the Black Friday shopping frenzy, Algolia soared to new heights, setting new records and delivering an ...

Bernadette Nixon

Chief Executive Officer and Board Member at Algolia

Generative AI’s impact on the ecommerce industry
ai

Generative AI’s impact on the ecommerce industry

When was your last online shopping trip, and how did it go? For consumers, it’s becoming arguably tougher to ...

Vincent Caruana

Senior Digital Marketing Manager, SEO

What’s the average ecommerce conversion rate and how does yours compare?
e-commerce

What’s the average ecommerce conversion rate and how does yours compare?

Have you put your blood, sweat, and tears into perfecting your online store, only to see your conversion rates stuck ...

Vincent Caruana

Senior Digital Marketing Manager, SEO

What are AI chatbots, how do they work, and how have they impacted ecommerce?
ai

What are AI chatbots, how do they work, and how have they impacted ecommerce?

“Hello, how can I help you today?”  This has to be the most tired, but nevertheless tried-and-true ...

Catherine Dee

Search and Discovery writer

Algolia named a leader in IDC MarketScape
algolia

Algolia named a leader in IDC MarketScape

We are proud to announce that Algolia was named a leader in the IDC Marketscape in the Worldwide General-Purpose ...

John Stewart

VP Corporate Marketing

Mastering the channel shift: How leading distributors provide excellent online buying experiences
e-commerce

Mastering the channel shift: How leading distributors provide excellent online buying experiences

Twice a year, B2B Online brings together America’s leading manufacturers and distributors to uncover learnings and industry trends. This ...

Jack Moberger

Director, Sales Enablement & B2B Practice Leader

Large language models (LLMs) vs generative AI: what’s the difference?
ai

Large language models (LLMs) vs generative AI: what’s the difference?

Generative AI and large language models (LLMs). These two cutting-edge AI technologies sound like totally different, incomparable things. One ...

Catherine Dee

Search and Discovery writer

What is generative AI and how does it work?
ai

What is generative AI and how does it work?

ChatGPT, Bing, Bard, YouChat, DALL-E, Jasper…chances are good you’re leveraging some version of generative artificial intelligence on ...

Catherine Dee

Search and Discovery writer

Feature Spotlight: Query Suggestions
product

Feature Spotlight: Query Suggestions

Your users are spoiled. They’re used to Google’s refined and convenient search interface, so they have high expectations ...

Jaden Baptista

Technical Writer

What does it take to build and train a large language model? An introduction
ai

What does it take to build and train a large language model? An introduction

Imagine if, as your final exam for a computer science class, you had to create a real-world large language ...

Vincent Caruana

Sr. SEO Web Digital Marketing Manager

The pros and cons of AI language models
ai

The pros and cons of AI language models

What do you think of the OpenAI ChatGPT app and AI language models? There’s lots going on: GPT-3 ...

Catherine Dee

Search and Discovery writer

How AI is transforming merchandising from reactive to proactive
e-commerce

How AI is transforming merchandising from reactive to proactive

In the fast-paced and dynamic realm of digital merchandising, being reactive to customer trends has been the norm. In ...

Lorna Rivera

Staff User Researcher

Top examples of some of the best large language models out there
ai

Top examples of some of the best large language models out there

You’re at a dinner party when the conversation takes a computer-science-y turn. Have you tried ChatGPT? What ...

Vincent Caruana

Sr. SEO Web Digital Marketing Manager

What are large language models?
ai

What are large language models?

It’s the era of Big Data, and super-sized language models are the latest stars. When it comes to ...

Catherine Dee

Search and Discovery writer

Mobile search done right: Common pitfalls and best practices
ux

Mobile search done right: Common pitfalls and best practices

Did you know that 86% of the global population uses a smartphone? The 7 billion devices connected to the Internet ...

Alexandre Collin

Staff SME Business & Optimization - UI/UX

Cloud Native meetup: Observability & Sustainability
engineering

Cloud Native meetup: Observability & Sustainability

The Cloud Native Foundation is known for being the organization behind Kubernetes and many other Cloud Native tools. To foster ...

Tim Carry

Algolia DocSearch is now free for all docs sites
product

Algolia DocSearch is now free for all docs sites

TL;DR Revamp your technical documentation search experience with DocSearch! Previously only available to open-source projects, we're excited ...

Shane Afsar

Senior Engineering Manager

Looking for something?

facebookfacebooklinkedinlinkedintwittertwittermailmail

In a world of artificial intelligence (AI), data serves as the foundation for machine learning (ML) models to identify trends and patterns, which it then uses to make predictions and decisions based on new, unseen data. The more data the model is trained on, the more accurate it can become in predicting outcomes or making decisions.

But just having a lot of data is not sufficient for training a good model. The saying “garbage in, garbage out” is a well-known concept in computing, indicating that flawed input data or instructions will generate flawed outputs.

Data quality concerns are frequently overlooked in ML research and education, with major textbooks focusing on the mathematical foundation of ML and using clean, organized, and pre-labeled “toy” datasets.

Despite this, implementing ML in a particular domain has to take into account that real-world data is flawed. This is a fact that any ML engineer or Data Scientist who works with productionalizing ML models is well-versed in, as most of the challenges in creating ML models that output quality results are data-related.

In this article, we will explore:

  1. Why do ML Models need a lot of data?
  2. Why is data quality important?
  3. The balance between data quantity and data quality

Why do ML Models need a lot of data?

Put simply, an ML model is a combination of a dataset and the algorithm used to train on that particular dataset. So, the same algorithm trained on different datasets will produce very different results.

An ML model does need a fair amount of examples from which it can learn. Depending on the   complexity of the problem  that they are trying to solve, machine learning models require different volumes of data, spanning from hundreds of data points for modeling a single user profile to millions of data points for language or computer vision models.

In general, the more complex the problem, the more data the model will need to learn and make accurate predictions. Additionally, if the data is noisy or contains many outliers, the model may require more data to filter out these anomalies.

When a model is trained on a limited amount of data, it may not have enough examples to accurately generalize to new data, resulting in overfitting or underfitting — basically the ML model learns the dataset “by heart” or fails to capture the underlying patterns in the data, resulting in poor performance when predictions are generated.

Why is data quality important?

Having more data is not always better, as the quality of the data is equally important. Poor quality data can negatively impact the performance of the model, even if there is a large amount of it.

The accuracy of the model’s predictions is highly dependent on the quality of the data it has been trained on. If the data is noisy, inconsistent, or contains errors, the model is likely to learn and propagate these errors, resulting in inaccurate predictions. As an example, consider a model that is trained to distinguish between pictures of cats and dogs. Having 5% of the data wrongly labeled as dogs when they are in fact cats will result in about a 5% error rate increase for that class on unseen data. Real data (be it collected or human annotated) can contain bugs and errors that may lead to uncertainty or mistakes at inference time.

To give another example: the ML architecture, data and techniques to train a model like ChatGPT have been public for a few years. However, a lot of engineering time went into producing a usable product. One key ingredient, or “secret sauce” if you will, has been the quality of the data: the curation of incorrect and duplicate data from the internet, plus the human annotated instruction/chat-like fine-tuning.

The balance between data quantity and data quality

In ML, there is often a trade-off between the quantity and quality of data. More data can lead to better performance of an ML model, but only if that data is of high quality meaning correct and diverse. On the other hand, even a small amount of high-quality data can produce a useful machine learning model, but only if the model is not too complex. For such cases, you can also use extrapolations to generate more data out of a small, quality dataset.

A few considerations to keep in mind when searching for the balance between the amount and quality of data:

  • Collecting and labeling a massive amount of data can be costly and time-consuming.
  • If the data is low quality, it may lead to a model with poor accuracy.
  • Data can be validated, cleaned and preprocessed to fix some errors like removing bad examples or filling missing values.
  • If you have a huge dataset, you don’t have to use all of it, as training a model with such a dataset is expensive. In fact, experimentation can be done — varying the dataset size to measure how much data is required to reach optimal performance.

Therefore, it is important to consider the specific task and context and determine the appropriate amount and quality of data required for building a successful machine learning model.

Where to start when collecting data for ML models

When going into data collection with the purpose of developing a ML model, start by asking yourself the following questions:

  • Is the data accurate and error free? Are we missing values or have incorrect values?
  • Is the data relevant? Is it linked to the problem we are trying to solve?
  • Is the data complete? Does it contain enough examples to train the machine learning model effectively?
  • Is the data consistent? Does it contain conflicting or contradictory information?
  • Does the data reflect a real-world scenario? 

The required volume of data depends on the complexity of the problem you are trying to solve, but if your dataset is less than a few thousand entries, a ML model might not be a good solution for your use case. Could the problem be solved using rules?

In addition, quality data is crucial for ensuring the accuracy and fairness of machine learning models. So plan to carefully curate, preprocess and validate it, thus ensuring it meets the necessary standards for the problem being solved.

At Algolia, customers can benefit from state-of-the-art AI search, while also benefiting from Algolia’s renowned performance, reliability and quality. All the AI training and management – from the selection of ML models, to their deployment, monitoring and optimization over time – is handled by Algolia.  Learn more about Algolia NeuralSearch.

About the author
Alexandra Anghel

Director of AI Engineering

linkedin

Recommended Articles

Powered byAlgolia Algolia Recommend

AI at scale: Managing ML models over time & across use cases
ai

Benoit Perrot

VP, Engineering

How to optimize an AI algorithm
ai

Rasit Abay

Senior Data Scientist

How to identify user search intent using AI and machine learning
ai

Ciprian Borodescu

AI Product Manager | On a mission to help people succeed through the use of AI