I'm creating an application which will programmatically sent HTTP GET and POST requests to a site and processing the data received.

If I want to release that application, do I have to get permission from the site's owners?


While I don't think there are any specific laws that would prevent you from doing this, personally I would be very sketched out about using someone else's web services that have not been officially published for the world.

The reason being is not legal reasons but the fact that you are taking your eggs and putting it into a basket that can be easily taken right from under you. What happens if you spend 3-12 man-months developing an app that is based on some specific web service and then the company discovers you and changes/locks down their API? Is all that time wasted? Do you have a backup plan?

Doing this is kind of like what Boxee tried to do with Hulu. Hulu has its own site but it clearly has streaming servers for its media content. Boxee tried to create their software so it could stream directly from Hulu, but Hulu had other plans. This never went to court but since Hulu can keep changing interfaces without any notice, it simply prevented Boxee from receiving its content.

So my advice is 1) use only public APIs that are guaranteed to stay public, 2) if not public, contact the company and see if you can enter into some kind of agreement or 3) look somewhere else for what you need, perhaps another open source project


In general, it's okay. Obviously, you should get your own legal advice, yada yada.

Web pages are crawled by all sorts of bots all the time, as any developer should know - most sites rely on it for search ranking, indexing, etc. Anything that is available without a user login or captcha is pretty much fair game. A robot can be expected to find anything that can be linked to from another site (i.e. does not require a logged-in session), and can't be expected to understand a Terms-of-Service document. That's why there's robots.txt.

To have the best chance of being on the right footing:

  • Use a distinct and obvious User-agent: header. Throwing in a contact address is excellent practice. FooCrawler 1.0 (webservices at example dot com)
  • Respect robots.txt
  • Minimise the load you put on their hosting providers. No polling every minute, put a few seconds delay between requests, etc.
  • Respect Cache-control and Expiry headers (if the HTTP response says it's valid for an hour, don't poll before then). If you're going to distribute your app widely and your users will 'target' the same sites, consider designing-in the facility to use a proxy to shift the load on to your own hosting.
  • Put in a reasonable effort to grok any copyright notices on the page
  • Understand (and document well in your code, if you do end up distributing it) that the services you use may break your app when they change, or deliberately block it.
  • If they block it, do not try to circumvent that

Have you checked the sites terms of service? Many prohibit just that sort of screen scraping.

  • So google crawlers read their licence agreements and skips them from index? I think it ain't happening. – Coder Dec 13 '11 at 21:21
  • You ain't google, your crawling is probably at best costing me bandwidth, at worst stealing customers. Google brings me customers. – Wyatt Barnett Dec 13 '11 at 21:26

Not the answer you're looking for? Browse other questions tagged or ask your own question.