At the risk of stating obvious tautologies, something is only illegal if there is a law against it. When someone puts up a website, it is considered open to the public by default. If there is content that should only be available to certain people, it's up to the web designer to secure it in some way.
When content is secured, and someone without authorization accesses it through hacking, this is generally considered morally equivalent to finding the door to somebody's house locked and then breaking in, and there are laws against doing so in most jurisdictions.
The intent of the website owner matters for a lot. A good deal of "deep web" content is content that site owners would like to make available, but isn't easily accessible to normal web crawlers. On the other hand, if an owner puts a rule in
robots.txt to exclude certain content from a crawler, and the crawler indexes it anyway, this is considered more or less equivalent to wandering onto someone's property when there's a NO TRESPASSING sign in clear view. But most websites welcome search engine traffic, because it drives actual users to the site, which helps accomplish the purpose of the site, whatever that purpose may be. (Usually making money, spreading information, or both.)
As for overuse of automated testing tools, this is something very different from a web crawler. A crawler has algorithms to only hit any given page once, and its principal purpose is to index sites so as to drive traffic to them, which most webmasters consider is worth the cost in bandwidth and processor power. But hitting the same page repeatedly by some tool that's not going to bring in new users does nothing to further the purposes of the site, and so the costs that it places on the site owner are essentially wasted. Unless the site owner actually asked for it, (for example, as part of a test of his site's capabilities,) it's generally considered unwelcome and harmful.