How to Choose a Web Scraping Partner

Recently we started our journey for a web scraping partner. While we considered building scrapers in house (and still might), we concluded that partnering with a DaaS (data as a service) solution would be quicker and more affordable in the short term. This blog post will explain the processes we went through, the questions we asked, the things we learned, the providers we interacted with, and how we eventually came to a decision on moving forward with a scraping partner.

Our Project: Scraping Millions of Ecommerce Product Pages

Our project was fairly simple in theory. We wanted to collect data from large retail websites and compare that data to find product matches

 

Initial Research

What does anyone do when they want to find anything these days? They Google it. So we did too. Here is a list of scraping services that we found through a few quick Google Searches. Here is a list of the ones we found along with our initial interactions with them.

https://www.octoparse.com/Pricing

webrobots.io – Thomas reached out to us quickly but said that they were booked up until the end of the year. We didn’t want to wait that long so we moved on.

 

https://www.parsehub.com/features

By |2018-12-04T01:11:54+00:00December 4th, 2018|Blog|0 Comments

About the Author:

Matt is a marketer and WordPress expert. Founder of Scepter, Matt oversees the larger web dev projects for the company. Seeing innovative ideas come to life is his passion, and therefore he pushes Scepter ahead to help our customers bring their projects to life.

Leave A Comment