I'm creating an application which will programmatically sent HTTP GET and POST requests to a site and processing the data received.
If I want to release that application, do I have to get permission from the site's owners?
Software Engineering Stack Exchange is a question and answer site for professionals, academics, and students working within the systems development life cycle. It only takes a minute to sign up.Sign up to join this community
While I don't think there are any specific laws that would prevent you from doing this, personally I would be very sketched out about using someone else's web services that have not been officially published for the world.
The reason being is not legal reasons but the fact that you are taking your eggs and putting it into a basket that can be easily taken right from under you. What happens if you spend 3-12 man-months developing an app that is based on some specific web service and then the company discovers you and changes/locks down their API? Is all that time wasted? Do you have a backup plan?
Doing this is kind of like what Boxee tried to do with Hulu. Hulu has its own site but it clearly has streaming servers for its media content. Boxee tried to create their software so it could stream directly from Hulu, but Hulu had other plans. This never went to court but since Hulu can keep changing interfaces without any notice, it simply prevented Boxee from receiving its content.
So my advice is 1) use only public APIs that are guaranteed to stay public, 2) if not public, contact the company and see if you can enter into some kind of agreement or 3) look somewhere else for what you need, perhaps another open source project
In general, it's okay. Obviously, you should get your own legal advice, yada yada.
Web pages are crawled by all sorts of bots all the time, as any developer should know - most sites rely on it for search ranking, indexing, etc. Anything that is available without a user login or captcha is pretty much fair game. A robot can be expected to find anything that can be linked to from another site (i.e. does not require a logged-in session), and can't be expected to understand a Terms-of-Service document. That's why there's
To have the best chance of being on the right footing:
User-agent:header. Throwing in a contact address is excellent practice.
FooCrawler 1.0 (webservices at example dot com)
Expiryheaders (if the HTTP response says it's valid for an hour, don't poll before then). If you're going to distribute your app widely and your users will 'target' the same sites, consider designing-in the facility to use a proxy to shift the load on to your own hosting.