People who code: we want your input. Take the Survey

Questions tagged [web-scraping]

Web scraping is automated information extraction from web sites.

Filter by
Sorted by
Tagged with
85
votes
7answers
19k views

How to be a good citizen when crawling web sites?

I'm going to be developing some functionality that will crawl various public web sites and process/aggregate the data on them. Nothing sinister like looking for e-mail addresses - in fact it's ...