I am working on a project that will scrape data from roughly about 6,000 websites. I am considering using a server/client model in which the server is responsible for scraping the data and importing them into a database while the client accesses the data.

To scrape the data, I am considering using a html parser like HtmlAgilityPack in which I will individually get the xpath of the data that I want scrapped per website. (This is a pain staking task, any better suggestions are welcome) These sites do not provide APIs however some provide RSS feeds for the data I want to scrape.

The "server" will scrape data from all the sites in a set interval (ie. minutes, hours) My question now is what should I consider in accomplishing this task programmatically?

  • From my approach of getting the xpath of the data from each website; I will probably end up having to create a class/function for each website. What are my alternatives?
  • How can I improve the performance of my program because obtaining the data in a timely manner is key as in using multi-threading techniques, asynchronous programming, memory usage, etc.
  • When the data scraped and before it is imported, it will be checked to see if it exists in the database. Is there a 'better' way instead of running a SQL query to loop through each node returned from the xpath to see if it exists?
  • This project will be written in C#, however I am open to other languages when it comes to the 'server'. Probably Python?
  • Once completed, ideally I will require a hosting service to run this program. What should I consider in that regard?

Bottom line: I am asking for the proper programming techniques to accomplish this project.

  • 1
    This is an off-site resource question. – DeadMG Jun 5 '16 at 10:03

If I were building a system like this from scratch, I would use Akka.Net to build a network of agents to do the scraping. Akka.Net will allow you to throw as many machines as you need at the problem, without re-architecting your application. You use the same techniques you would use in any other program to improve its performance, but Akka will relieve you of most of the concurrency pain.

You need one class per scraper; every website is different. Factor out everything that is common to all scrapers into its own class.

You should create a Data Transfer Object (DTO) for each site, if the data you're returning differs for each site, or a single, universal DTO for all the sites. Each scraper should accept whatever parameters it needs to perform a search on a particular site, scrape whatever pages are required, and return the resulting DTO. You can then save that DTO to the database at your convenience.

Once you get some experience with HTML Agility Pack, you will find that you will rely less on several XPaths, and more on one or two Xpaths that will get you in the ballpark, with subsequent SelectAllNodes() or SelectSingleNode() calls with a simple selector like tr to get you the rest of the way. Whatever you do, don't store the XPaths in a database or configuration file; just write them directly into the scraper class.

You should figure out how to schedule the scrapers so that you can space requests directed at the same site one minute apart.

Your hosting problem is unremarkable. You need a host for your client page; the scrapers just need an Internet connection. Be prepared to deal with things like captcha's, proxys and the like. Most websites are not well-written.

Finally, a word of advice: make sure your business model is sustainable. Most websites do not take kindly to wholesale copying of their entire website or data store, regardless of how you acquire it. It will be better for you if your business model only requires bits of information from each site, and not the whole thing.

  • I really appreciate your input!! I will definitely check out Akka.net, after looking at the site and documentation it is looks promising. When I was testing this on a small number of sites, I was using DTOs but didn't know the term for them until now so thanks for that! (Will do some more googling to improve them). You are right about HAP, as of right now I feel like some sites are easier to obtain the nodes with the xpath than others so after 6k sites I should get the gist. I do not plan on storing the xpaths but out of curiosity why should I encode them? – NuWin Jun 5 '16 at 0:37
  • In regards to the hosting, your totally right all the server needs is an Internet connection, however if this were to be commercialized, it's ideal for a server host. I appreciate your final words and I am totally aware of the legal ramifications however the data that I'm obtaining is small from each site and I would think that they wouldn't mind since it is public information. – NuWin Jun 5 '16 at 0:40
  • re: encoding... Sorry, poor choice of words. I modified my answer accordingly. – Robert Harvey Jun 5 '16 at 4:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.