Web crawling is having a software program (a "bot") systematically explore websites and index the data it finds, making it easy for people to locate by using a search engine.
Web scraping, a slightly different form of gathering web data, involves collecting (downloading) specific types of information, for instance, about pricing.
In ecommerce, both of these types of data gathering are especially valuable because the data collected and analyzed can lead to marketers data-based decisions that can boost sales.
Marketers can compare data about products being sold on other sites with the same ones they're selling, for instance.
If they find out that shoppers are routinely entering certain keywords in a search engine to locate a given product, they might decide to add those words to the product description to attract potential buyers to the product listing.
Consumers typically want the best deals, and they can easily search for the lowest prices on the web. If a company sees that a competitor has a lower price on a product they offer, they can lower their own price to ensure that prospective customers won't choose the competitor's due solely to a lower cost.
By gathering product review and ranking data, marketers and businesspeople can uncover information about flaws in their own and competitors' products.
They can also use crawler technology to monitor product reviews and rankings so that they can swiftly respond when people post negative comments, thereby improving their customer service.
They can find out which products are bestsellers and potentially identify hot new markets.
All of this revenue-impacting activity makes ecommerce web crawling and web scraping an important and lucrative subdomain of these activities as a whole.