1

My Title clearly indicated my lack of understanding about the core concept for pattern matching - specifically using .HTACCESS to block Bad-Bots from accessing a site, which they do in order to crawl or mirror copy, consequently using up bandwidth.

But my question isn't about .HTACCESS - it's about the obvious ineffectiveness (IMO) of using a long list for searching and matching.

Isn't it far better to Allow a list of positive, as opposed to Match against list of negatives??

For example:

1st explanation attempt:

If the User Agent does NOT match one of these Good-Bots, then block.

If the User Agent matches one of these Bad-Bots, then block.

I can't express this idea by formula or algorithm because I don't know how to - assuming it could be, but I suppose if I could express the idea it would be . . .

2nd explanation attempt:

If THIS is NOT 'A' ('A' being a list of positives), then deny.

If THIS is ONE INSTANCE of 'A' ('A' being a list of negatives), then deny.

3rd try!

If THIS is not RED, then deny.

If this is BLUE, YELLOW, GREEN, (i.e. NOT RED), then deny.

If this makes sense, why would a web developer use the latter approach if the latter's list is more than the former. Presumably there are less Good-Bots (User Agents) than there are Bad-Bots (ignoring the fact the UA can be forged.)?

Ultimately, wouldn't it be far better to create an index of all common Good-Bots and use this to search and match, rather than list a seemingly and potentially infinite list of Bad-Bots? (Not forgetting the time it would take to update such a list with new Bad-Bots.)

Reference: perishablepress.com/4g-ultimate-user-agent-blacklist/

Why search & match for MORE negatives, than search for less positives?

7
  • That only makes sense if you're willing to white-list the good bots. It might work for bots, but not for email, since some people have a very large white list. In a role-based-permissions system, white lists are easier, but for some systems like Stack Overflow, white-listing is impractical. So it depends. Mar 23, 2015 at 23:10
  • 2
    I think you can cut most of the explanatory text from this question by simply using the terms "whitelist" and "blacklist", since your actual question appears to be "Why does .HTACCESS use a blacklist rather than a whitelist?"
    – Ixrec
    Mar 23, 2015 at 23:12
  • 1
    But it does have a whitelist too. deny from all /newline/ allow from xx.xxx.xx.xx /newline/ allow from xx.xxx.xx.xx ... Its just the matter of what order you specify it in.
    – user40980
    Mar 23, 2015 at 23:15
  • 1
    I see. It's a balance of sorts. Depending on the major stakeholder, as it were. So Stack Overflow's approach is: IF in blacklist then DENY, rather than IF in White-list then DENY. The idea has left my mind, but it seems that the less variables (individual amounts) there are, the "easier" is it to search and match the lesser, such as @MichaelT point about allowing IP's - but for an undefined amount (Good IPs), it's better to search & match for Bad-IPs. I'll have to read some more. "If NOT in Good-Bot White-list, then DENY." Comparing against the properties of an Index as opposed to search.
    – user172258
    Mar 23, 2015 at 23:30
  • 1
    @DThomas I will point out that with with some hackery of apache rewrite rules, using the extended redirect you could have an external process do the rewrite allowing for, well, whatever you want. You could also add a custom handler or play with the environment settings to mod_authz_host allowing you to do even more. Lots of flexibility available if there are issues with the current approach.
    – user40980
    Mar 23, 2015 at 23:46

1 Answer 1

1

This is a matter of what you want the default to be.

If you want to allow by default, you need to list the blocked ones (blacklist). If you want to deny by default, you need to list the allowed ones (whitelist).

The User-Agent strings are incredibly varied. If you used white-list, your site wouldn't work for some people and that's something you absolutely don't want. So you just selectively blacklist the bots that actually cause excessive traffic.

That applies to public-facing webs. Intranet sites and sites for specific customers may, and often do, whitelist just the legitimate users.

Presumably there are less Good-Bots (User Agents) than there are Bad-Bots (ignoring the fact the UA can be forged.)?

No, it certainly isn't. There are many niche browsers and various scripts and tools and the browsers include various bits of configuration and you can't ever hope to collect the complete set, because new ones are being created every day.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy