I am working on a project that will scrape data from roughly about 6,000 websites. I am considering using a server/client model in which the server is responsible for scraping the data and importing them into a database while the client accesses the data.
To scrape the data, I am considering using a html parser like HtmlAgilityPack in which I will individually get the xpath of the data that I want scrapped per website. (This is a pain staking task, any better suggestions are welcome) These sites do not provide APIs however some provide RSS feeds for the data I want to scrape.
The "server" will scrape data from all the sites in a set interval (ie. minutes, hours) My question now is what should I consider in accomplishing this task programmatically?
- From my approach of getting the xpath of the data from each website; I will probably end up having to create a class/function for each website. What are my alternatives?
- How can I improve the performance of my program because obtaining the data in a timely manner is key as in using multi-threading techniques, asynchronous programming, memory usage, etc.
- When the data scraped and before it is imported, it will be checked to see if it exists in the database. Is there a 'better' way instead of running a SQL query to loop through each node returned from the xpath to see if it exists?
- This project will be written in C#, however I am open to other languages when it comes to the 'server'. Probably Python?
- Once completed, ideally I will require a hosting service to run this program. What should I consider in that regard?
Bottom line: I am asking for the proper programming techniques to accomplish this project.