I am trying to build a system for my company which wants to check for unusual/abusive pattern of users (mainly web scrapers).

Currently the logic I have implemented parses the http access logs and takes into account the following parameters to calculate the potential of a user being a scraper or bot:

  1. It checks v/s HTTP 'POST/GET' requests ratio for each IP

  2. It calculates the ratio of unique URLs and total number of hits (sparsity) by each IP

Based on the above two parameters, we try to block any IP showing unusual behaviour, but these two parameters alone have not been sufficient for bot detection. Thus I would like to know:

  1. Are there any other parameters which can be included to improve the detection?

  2. I found a paper published in ACM library which follows the Bayesian approach to detect a crawler. Has anyone used this? How effective is this?

  3. Stack Overflow and other high traffic sites have such kind of systems deployed, what logic do they follow to keep unwanted spammers/crawlers away in real time?

  • Hmmm... You'll need to further expand on what you consider unusual/abusive pattern...
    – yannis
    Dec 23, 2011 at 6:32
  • Uuusual or abusive pattern would be if a user is crawling my data hiding behind common user agents, using my bandwidth and in return is not giving me anything. Dec 23, 2011 at 6:41
  • @LokiAstari Why will a crawler with malicious intent follow robots.txt? Dec 23, 2011 at 7:44
  • @LokiAstari Whitelisting will take care of good bots, but then question is not good v/s bod bots at the first level, its identification of real users v/s bots to begin with, search bots like google-bot, msnbot will have their IPs as whitelisted. Dec 23, 2011 at 7:54
  • 2
    Possible duplicate of stackoverflow.com/questions/233192/…
    – Mike Nakis
    Dec 23, 2011 at 8:25

4 Answers 4


What are you trying to protect against? Is the concern that the bot will use excessive bandwidth or that they will get a copy of all your website content?

In either case an analysis of the log file after the fact will do nothing to prevent either. If you are concerned with someone stealing your content, what good does it do know that someone just did it last night? A little like locking the door after you have been robbed.

Much better to simply implement bandwidth throttling, simply limit the number of pages per unit time (minute/hour whatever) that you website will deliver to a specific IP address, or better still a block of IP addresses.

Remember that someone trying to steal your content may be very clever. The most likely will use multiple IP addresses.

Also be aware that there are hardware appliances that can be installed in a data center to do this in real-time.

  • Trying to achieve both the above mentioned things. My script runs after half an hour and analyses the data of previous half an hour, thus bot is detected almost in real time. Throttling bandwidth doesnt solve the problem if a crawler is crawling at a very slow rate(remember I still want to protect my data). The solution I have implemented so far has 90% accuracy rate(tested on a week s data on multiple domains with real traffic). Dec 23, 2011 at 14:14
  • Having a multiple IPs does not make a difference as each IP will behave in a manner different than real user. And Lastly, Hardware solutions are expensive and have limitations. Dec 23, 2011 at 14:15
  • bilkulbekar: you state you want to protect your data. How many requests would one need to make against your site to get all the data? Lets say it is 1,000 pages. If I use 100 IP addresses and each sends you 10 requests with random delays between 1 & 3 min I can get your whole site in less than 1/2 hour.
    – JonnyBoats
    Dec 23, 2011 at 14:29
  • If the problem is bots slurping resources of one kind or another, better than banning the bots is to avoid the "attractive nuisance" of links altogether. use a javascript "onclick" action so there is no apparent link at all.
    – ddyer
    Dec 23, 2011 at 20:02
  • 1
    @JonnyBoats: Well, a couple of the domains on which I have deployed my solution has over 3 million pages. And also if you are crawling at a rate of 1 request per 1-3 minutes, even with 100 IPs, all the IPs will be detected as crawlers after a certain period of time. Dec 26, 2011 at 10:05

Just embed some invisible links in your html. Anyone that follows one is a robot or scraper.

  • 1
    ddyer: This trick is already known, and doesnt work in all the use cases. The problem is crawler may not decide to crawl only specific parts of the site, and not the complete site, in that case multiple invisible links will have to be setup. Dec 23, 2011 at 6:56
  • Also, the site has search option, which returns a set of listings(the number being in millions), bot can try to crawl this data sequentially thus using a lot of bandwidth, but again not hitting any of the invisible link embedded. Dec 23, 2011 at 6:59
  • @YannisRizos When I said its known, I did not mean to the enemy. :) Dec 23, 2011 at 7:05
  • @bilkulbekar :)
    – yannis
    Dec 23, 2011 at 7:33
  • @bil this is still a very good idea as a honeypot, and should be implemented -- anything hitting the invisible-via-CSS links has to be a bot parsing the raw HTML. Dec 23, 2011 at 23:18

Examine the frequency at which requests are coming in, and if it is exceedingly high, throttle the requests. This way, you are not blocking anyone, and yet nobody can consume too much bandwidth.

  • The Bayesian Approach I have mentioned uses this, but I am concerned about the resource consumption if this is applied in real time. A script which parses the apache access logs and gives out the results looks like more convenient. Dec 23, 2011 at 9:23
  • 1
    Yes, but will such a script be running concurrently so as to be able to provide throttling advice in real time? Because blacklisting is a much too drastic measure and there will inevitably be false positives.
    – Mike Nakis
    Dec 23, 2011 at 9:37
  • Its a good approach, but defining 'exceedingly high' is not as clear as it looks, the system will have to be trained for this. Dec 23, 2011 at 11:44
  • 1
    Since you will only be looking at page accesses, (no css, no script, no images) it should not be rocket science, either. And since you will not actually be blocking anyone, it does not really matter if you overshoot while training.
    – Mike Nakis
    Dec 23, 2011 at 17:17

I like requests-per-session-per-second, sessions-per-IP, and request pace over time.

The first - requests-per-session-per-second - will almost invaribly be different between humans and bots.

The second - sessions-per-IP - might appear to be easy to do, but you probably won't be able to tell the difference between a large number of users behind a NAT/firewall -and- a multithreaded bot. It's probably a good "additonal indicator", however.

The third - request pace over time - requires a little explaining. Bots tend to have their own analysis pace, processing "lag", and turnaround time between page requests. Depending upon what they're doing, a bot can retrieve and parse tens-of-kilobytes of webpage content without flinching, and turn around and make yet another request. However, this doesn't differ from what a human might do when, say, they immediately see a link they want, and click on it before the rest of the page loads.

However, a human - even one that frequently visits your site - will likely only remember how to quickly navigate the first few levels of your site using this method. After a few levels, the human will likely "slow down", and read more content/take more time to process what they've requested. A bot, on the other hand, will continue at their original pace throughout its entire interaction with your site.

Based upon this, I'd say any session that quickly (more-than-humanly-possible?) processes the retrieved content should be initially categorized a bot, but not cut off. If, after two, perhaps three levels of navigation into your site the session still continues to make "faster-than-human" requests, definitively call it a bot, and cut it off.

If a human can actually achieve such a high and sustained interaction with your site, you probably have to redesign your site anyway (lol), and either give the user shortcuts to deep portions of your site, or "flatten" your site altogether.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.