Search by Algolia
What’s the secret of online merchandise management? Giving store merchandisers the right tools
e-commerce

What’s the secret of online merchandise management? Giving store merchandisers the right tools

As a successful in-store boutique manager in 1994, you might have had your merchandisers adorn your street-facing storefront ...

Catherine Dee

Search and Discovery writer

New features and capabilities in Algolia InstantSearch
engineering

New features and capabilities in Algolia InstantSearch

At Algolia, our business is more than search and discovery, it’s the continuous improvement of site search. If you ...

Haroen Viaene

JavaScript Library Developer

Feature Spotlight: Analytics
product

Feature Spotlight: Analytics

Analytics brings math and data into the otherwise very subjective world of ecommerce. It helps companies quantify how well their ...

Jaden Baptista

Technical Writer

What is clustering?
ai

What is clustering?

Amid all the momentous developments in the generative AI data space, are you a data scientist struggling to make sense ...

Vincent Caruana

Sr. SEO Web Digital Marketing Manager

What is a vector database?
product

What is a vector database?

Fashion ideas for guest aunt informal summer wedding Funny movie to get my bored high-schoolers off their addictive gaming ...

Vincent Caruana

Sr. SEO Web Digital Marketing Manager

Unlock the power of image-based recommendation with Algolia’s LookingSimilar
engineering

Unlock the power of image-based recommendation with Algolia’s LookingSimilar

Imagine you're visiting an online art gallery and a specific painting catches your eye. You'd like to find ...

Raed Chammam

Senior Software Engineer

Empowering Change: Algolia's Global Giving Days Impact Report
algolia

Empowering Change: Algolia's Global Giving Days Impact Report

At Algolia, our commitment to making a positive impact extends far beyond the digital landscape. We believe in the power ...

Amy Ciba

Senior Manager, People Success

Retail personalization: Give your ecommerce customers the tailored shopping experiences they expect and deserve
e-commerce

Retail personalization: Give your ecommerce customers the tailored shopping experiences they expect and deserve

In today’s post-pandemic-yet-still-super-competitive retail landscape, gaining, keeping, and converting ecommerce customers is no easy ...

Vincent Caruana

Sr. SEO Web Digital Marketing Manager

Algolia x eTail | A busy few days in Boston
algolia

Algolia x eTail | A busy few days in Boston

There are few atmospheres as unique as that of a conference exhibit hall: the air always filled with an indescribable ...

Marissa Wharton

Marketing Content Manager

What are vectors and how do they apply to machine learning?
ai

What are vectors and how do they apply to machine learning?

To consider the question of what vectors are, it helps to be a mathematician, or at least someone who’s ...

Catherine Dee

Search and Discovery writer

Why imports are important in JS
engineering

Why imports are important in JS

My first foray into programming was writing Python on a Raspberry Pi to flicker some LED lights — it wasn’t ...

Jaden Baptista

Technical Writer

What is ecommerce? The complete guide
e-commerce

What is ecommerce? The complete guide

How well do you know the world of modern ecommerce?  With retail ecommerce sales having exceeded $5.7 trillion worldwide ...

Vincent Caruana

Sr. SEO Web Digital Marketing Manager

Data is king: The role of data capture and integrity in embracing AI
ai

Data is king: The role of data capture and integrity in embracing AI

In a world of artificial intelligence (AI), data serves as the foundation for machine learning (ML) models to identify trends ...

Alexandra Anghel

Director of AI Engineering

What are data privacy and data security? Why are they  critical for an organization?
product

What are data privacy and data security? Why are they critical for an organization?

Imagine you’re a leading healthcare provider that performs extensive data collection as part of your patient management. You’re ...

Catherine Dee

Search and Discovery writer

Achieving digital excellence: Algolia's insights from the GDS Retail Digital Summit
e-commerce

Achieving digital excellence: Algolia's insights from the GDS Retail Digital Summit

In an era where customer experience reigns supreme, achieving digital excellence is a worthy goal for retail leaders. But what ...

Marissa Wharton

Marketing Content Manager

AI at scale: Managing ML models over time & across use cases
ai

AI at scale: Managing ML models over time & across use cases

Just a few years ago it would have required considerable resources to build a new AI service from scratch. Of ...

Benoit Perrot

VP, Engineering

How continuous learning lets machine learning  provide increasingly accurate predictions and recommendations
ai

How continuous learning lets machine learning provide increasingly accurate predictions and recommendations

What new data points have you learned lately? Learning is never ending (hence the phrase “lifelong learning”), so chances are ...

Vincent Caruana

Sr. SEO Web Digital Marketing Manager

Using personalization to boost ecommerce ROI: trends, facts, tips
e-commerce

Using personalization to boost ecommerce ROI: trends, facts, tips

You love blue. You’ve bought several blue items from a particular up-and-coming brand. You’re also a ...

Catherine Dee

Search and Discovery writer

Looking for something?

facebookfacebooklinkedinlinkedintwittertwittermailmail

In a world of artificial intelligence (AI), data serves as the foundation for machine learning (ML) models to identify trends and patterns, which it then uses to make predictions and decisions based on new, unseen data. The more data the model is trained on, the more accurate it can become in predicting outcomes or making decisions.

But just having a lot of data is not sufficient for training a good model. The saying “garbage in, garbage out” is a well-known concept in computing, indicating that flawed input data or instructions will generate flawed outputs.

Data quality concerns are frequently overlooked in ML research and education, with major textbooks focusing on the mathematical foundation of ML and using clean, organized, and pre-labeled “toy” datasets.

Despite this, implementing ML in a particular domain has to take into account that real-world data is flawed. This is a fact that any ML engineer or Data Scientist who works with productionalizing ML models is well-versed in, as most of the challenges in creating ML models that output quality results are data-related.

In this article, we will explore:

  1. Why do ML Models need a lot of data?
  2. Why is data quality important?
  3. The balance between data quantity and data quality

Why do ML Models need a lot of data?

Put simply, an ML model is a combination of a dataset and the algorithm used to train on that particular dataset. So, the same algorithm trained on different datasets will produce very different results.

An ML model does need a fair amount of examples from which it can learn. Depending on the   complexity of the problem  that they are trying to solve, machine learning models require different volumes of data, spanning from hundreds of data points for modeling a single user profile to millions of data points for language or computer vision models.

In general, the more complex the problem, the more data the model will need to learn and make accurate predictions. Additionally, if the data is noisy or contains many outliers, the model may require more data to filter out these anomalies.

When a model is trained on a limited amount of data, it may not have enough examples to accurately generalize to new data, resulting in overfitting or underfitting — basically the ML model learns the dataset “by heart” or fails to capture the underlying patterns in the data, resulting in poor performance when predictions are generated.

Why is data quality important?

Having more data is not always better, as the quality of the data is equally important. Poor quality data can negatively impact the performance of the model, even if there is a large amount of it.

The accuracy of the model’s predictions is highly dependent on the quality of the data it has been trained on. If the data is noisy, inconsistent, or contains errors, the model is likely to learn and propagate these errors, resulting in inaccurate predictions. As an example, consider a model that is trained to distinguish between pictures of cats and dogs. Having 5% of the data wrongly labeled as dogs when they are in fact cats will result in about a 5% error rate increase for that class on unseen data. Real data (be it collected or human annotated) can contain bugs and errors that may lead to uncertainty or mistakes at inference time.

To give another example: the ML architecture, data and techniques to train a model like ChatGPT have been public for a few years. However, a lot of engineering time went into producing a usable product. One key ingredient, or “secret sauce” if you will, has been the quality of the data: the curation of incorrect and duplicate data from the internet, plus the human annotated instruction/chat-like fine-tuning.

The balance between data quantity and data quality

In ML, there is often a trade-off between the quantity and quality of data. More data can lead to better performance of an ML model, but only if that data is of high quality meaning correct and diverse. On the other hand, even a small amount of high-quality data can produce a useful machine learning model, but only if the model is not too complex. For such cases, you can also use extrapolations to generate more data out of a small, quality dataset.

A few considerations to keep in mind when searching for the balance between the amount and quality of data:

  • Collecting and labeling a massive amount of data can be costly and time-consuming.
  • If the data is low quality, it may lead to a model with poor accuracy.
  • Data can be validated, cleaned and preprocessed to fix some errors like removing bad examples or filling missing values.
  • If you have a huge dataset, you don’t have to use all of it, as training a model with such a dataset is expensive. In fact, experimentation can be done — varying the dataset size to measure how much data is required to reach optimal performance.

Therefore, it is important to consider the specific task and context and determine the appropriate amount and quality of data required for building a successful machine learning model.

Where to start when collecting data for ML models

When going into data collection with the purpose of developing a ML model, start by asking yourself the following questions:

  • Is the data accurate and error free? Are we missing values or have incorrect values?
  • Is the data relevant? Is it linked to the problem we are trying to solve?
  • Is the data complete? Does it contain enough examples to train the machine learning model effectively?
  • Is the data consistent? Does it contain conflicting or contradictory information?
  • Does the data reflect a real-world scenario? 

The required volume of data depends on the complexity of the problem you are trying to solve, but if your dataset is less than a few thousand entries, a ML model might not be a good solution for your use case. Could the problem be solved using rules?

In addition, quality data is crucial for ensuring the accuracy and fairness of machine learning models. So plan to carefully curate, preprocess and validate it, thus ensuring it meets the necessary standards for the problem being solved.

At Algolia, customers can benefit from state-of-the-art AI search, while also benefiting from Algolia’s renowned performance, reliability and quality. All the AI training and management – from the selection of ML models, to their deployment, monitoring and optimization over time – is handled by Algolia.  Learn more about Algolia NeuralSearch.

About the author
Alexandra Anghel

Director of AI Engineering

linkedin

Recommended Articles

Powered byAlgolia Algolia Recommend

AI at scale: Managing ML models over time & across use cases
ai

Benoit Perrot

VP, Engineering

How to optimize an AI algorithm
ai

Rasit Abay

Senior Data Scientist

How to identify user search intent using AI and machine learning
ai

Ciprian Borodescu

AI Product Manager | On a mission to help people succeed through the use of AI