Recently we started our journey for a web scraping partner. While we considered building scrapers in house (and still might), we concluded that partnering with a DaaS (data as a service) solution would be quicker and more affordable in the short term. This blog post will explain the processes we went through, the questions we asked, the things we learned, the providers we interacted with, and how we eventually came to a decision on moving forward with a scraping partner.

Our Project: Scraping Millions of Ecommerce Product Pages

Our project was fairly simple in theory. We wanted to collect data from large retail websites and compare that data to find product matches

 

Initial Research

What does anyone do when they want to find anything these days? They Google it. So we did too. Here is a list of scraping services that we found through a few quick Google Searches. Here is a list of the ones we found along with our initial interactions with them.

https://www.octoparse.com/Pricing

webrobots.io – Thomas reached out to us quickly but said that they were booked up until the end of the year. We didn’t want to wait that long so we moved on.

 

https://www.parsehub.com/features