I have my own website that displays some data I get from webscraping. I run the webscraping script on my website host server. I just realized that maybe I'm accessing other websites too quickly through this script. For example, I just wrote a script that accessed 15 websites on the same domain in about a second. My own website server crashed shortly thereafter. My webhost was having some other problems so I don't know if these two events are related or not.
But, I guess it brought up another question. Do I need to limit how fast I access websites to prevent a denial of service (DOS) type warning from either the website I'm accessing or my own webserver host? Is there some sort of safe rate of websites per second or per minute I should follow? Just wondering,
EDIT: To clarify, yes I could slow down the collection process but I'm actually scraping a lot of small pieces of information from different sites so accessing a lot of websites might be unavoidable. Also, based on what I'm doing, getting the information as quickly as possible would be beneficial. So I can slow the process down but I guess I'm looking for how much I can "push" the limit to get the information in the optimal time.