1

I have my own website that displays some data I get from webscraping. I run the webscraping script on my website host server. I just realized that maybe I'm accessing other websites too quickly through this script. For example, I just wrote a script that accessed 15 websites on the same domain in about a second. My own website server crashed shortly thereafter. My webhost was having some other problems so I don't know if these two events are related or not.

But, I guess it brought up another question. Do I need to limit how fast I access websites to prevent a denial of service (DOS) type warning from either the website I'm accessing or my own webserver host? Is there some sort of safe rate of websites per second or per minute I should follow? Just wondering,

EDIT: To clarify, yes I could slow down the collection process but I'm actually scraping a lot of small pieces of information from different sites so accessing a lot of websites might be unavoidable. Also, based on what I'm doing, getting the information as quickly as possible would be beneficial. So I can slow the process down but I guess I'm looking for how much I can "push" the limit to get the information in the optimal time.

Improve this question
6
  • 4
    Yes you do, no there isn't. Be nice.
    – user40980
    Dec 10, 2015 at 16:43
  • 1
    If you are running it from your server then it is not a DDOS. It is not distributed if is it is coming from one site. Hitting 15 sites in a second should should not be crashing a server.
    – paparazzo
    Dec 10, 2015 at 16:51
  • Think also about how often the data changes. If you're collecting weather information, polling every 5 to 10 minutes is probably sufficient.
    – Dan Pichelman
    Dec 10, 2015 at 17:00
  • 1
    It depends a lot on the server. You can hit google.com at a far higher rate of frequency then grandpa's homepage.
    – Gort the Robot
    Dec 10, 2015 at 18:22
  • 2
    How often do you need the data? (What creates the demand?) Are you trying to see every change or just be more or less up to date?
    – Erik Eidt
    Dec 10, 2015 at 18:30

2 Answers 2

Reset to default
2

There is no well defined limits, but by looking into the access log of my http://gcc-melt.org/ site (which is quite specialized and has only about a thousand hits per day, including robots), I see that well behaved bots:

  • are querying the robots.txt file and following it

  • are making requests spaced by a few seconds (typically 5) at least

I would advise your program to behave likewise. Don't query the same website more than once every few seconds (in a rate similar to what humans are browsing). Of course your webscraper can make requests to other sites before that.

If you are querying (programmatically) some particular website very often (e.g. more than once per second), I would suggest to at least try to contact (by email) its human webmaster. Some webmasters are blocking requests from hosts which query them too often.

Remember that some web sites (like mines) are paid by individuals, and their hosting contract may put some limits (typically on bandwidth or monthly data volume). So you should be nice with them.

Improve this answer
1

Yes you do, no there isn't. Be nice. --MichaelT

As @MichaelT says there is no set rule, but how often does the information on the foreign sites that you scrape change? You might cache the results locally on your server (or some DB) and only update/rescrape every minute, hour, day etc.

For example, movie showtimes change slowly, only recheck them every hour. A basketball game score changes often, you need to check every few seconds.

Improve this answer
2
  • 1
    added edit comment above but in case you might miss it: To clarify, yes I could slow down the collection process but I'm actually scraping a lot of small pieces of information from different sites so accessing a lot of websites might be unavoidable. Also, based on what I'm doing, getting the information as quickly as possible would be beneficial. So I can slow the process down but I guess I'm looking for how much I can "push" the limit to get the information in the optimal time.
    – bluemoon12
    Dec 10, 2015 at 17:14
  • Caching recent results locally won't slow you down - in fact, it speeds you up because it saves the whole HTTP then scrape process. But there is a cost that data may become a little stale. If the user sees data that is one minute out of date, is that a serious problem??? If it is movie showtimes, nobody would care. If it is a sports score, they might. (I edited my answer to include these examples)
    – user949300
    Dec 11, 2015 at 2:42

Not the answer you're looking for? Browse other questions tagged or ask your own question.