• When Google, Bing or Yahoo are crawling content from Web sites, what makes it legal? Is there a public registry of only allowed crawlers?
  • When researchers are crawling Deep Web, what makes their efforts legal?
  • When tester automate their tests with Selenium or JMeter and hit same site multiple times, what makes their effort illegal?

In each of those cases, an automate is consuming Internet bandwidth of the Web site, and copying their content. But some are considered legal, and other are not.

  • 4
    Which legal jurisdiction(s) are you referring to? – Greg Hewgill Apr 27 '14 at 20:31
  • Is "going against the website/provider/... terms of service" what you call illegal? There is quite a step between the two concepts. – user44761 Apr 27 '14 at 20:37
  • @Tibo: There should be quite a step. In the USA, there often isn't, due to the Computer Fraud and Abuse Act. If that sounds strange, well, there's a reason a lot of people think that particular law needs some serious reform. – Mason Wheeler Apr 27 '14 at 20:42
  • Counter-question: What makes it legal for the owner of the website to block your IP? – Robert Harvey Apr 27 '14 at 20:50
  • 5
    This question appears to be off-topic because it is not about software development. – GrandmasterB Apr 27 '14 at 21:14

At the risk of stating obvious tautologies, something is only illegal if there is a law against it. When someone puts up a website, it is considered open to the public by default. If there is content that should only be available to certain people, it's up to the web designer to secure it in some way.

When content is secured, and someone without authorization accesses it through hacking, this is generally considered morally equivalent to finding the door to somebody's house locked and then breaking in, and there are laws against doing so in most jurisdictions.

The intent of the website owner matters for a lot. A good deal of "deep web" content is content that site owners would like to make available, but isn't easily accessible to normal web crawlers. On the other hand, if an owner puts a rule in robots.txt to exclude certain content from a crawler, and the crawler indexes it anyway, this is considered more or less equivalent to wandering onto someone's property when there's a NO TRESPASSING sign in clear view. But most websites welcome search engine traffic, because it drives actual users to the site, which helps accomplish the purpose of the site, whatever that purpose may be. (Usually making money, spreading information, or both.)

As for overuse of automated testing tools, this is something very different from a web crawler. A crawler has algorithms to only hit any given page once, and its principal purpose is to index sites so as to drive traffic to them, which most webmasters consider is worth the cost in bandwidth and processor power. But hitting the same page repeatedly by some tool that's not going to bring in new users does nothing to further the purposes of the site, and so the costs that it places on the site owner are essentially wasted. Unless the site owner actually asked for it, (for example, as part of a test of his site's capabilities,) it's generally considered unwelcome and harmful.


I didn't know automated tests that hit the same site multiple times is illegal, but it does make sense if you think about it.

I think it is because it can be included in the Denial Of Service realm if you hit it very frequently (e.g. multiple times a second for "testing" purposes).

Crawling the internet means just that: crawling. You don't hit a page multiple times a minute since you would waste precious resources on nothing (a page would not change that often).

  • When tester automate their tests with Selenium or JMeter and hit same site multiple times, what makes their effort illegal?

When a tester hits the same (external site that doesn't belong to him) with multiple test, he just made a bad test. That stuff should mocked.

  • 1
    That not true. In contrast to unit and integration tests, end-to-end tests may require interaction with external systems as well. Mocking external system/Web site may not reveal all integration problems. Obviously, running the test hitting external site should require some agreement with an owner of that site. Anyway, -1, because this does not answer the question. – dzieciou Apr 28 '14 at 4:21

Not the answer you're looking for? Browse other questions tagged or ask your own question.