Validating allowed characters or validating disallowed characters

Question

I've always validated my user input based on a list of valid/allowed characters, rather than a list of invalid/disallowed characters (or simply no validation). It's just a habit I picked up, probably on this site and I've never really questioned it until now.

It makes sense if you wish to, say, validate a phone number, or validate an area code, however recently I've realised I'm also validating input such as Bio Text fields, User Comments, etc. for which the input has no solid syntax.

The main advantage has always seemed to be: Validating allowed chars reduces the risk of you missing a potentially malicious character, but increases the risk the of you not allowing a character which the user may want to use. The former is more important.

But, providing I am correctly preventing SQL Injection (with prepared statements) and also escaping output, is there any need for this extra barrier of protection? It seems to me as if I am just allowing practically every character on the keyboard, and am forgetting to allow some common characters.

Is there an accepted practice for this situation? Or am I missing something obvious?

Thanks.

Answer 1

Any implementation trying to detect "malicious characters" is flawed, when you look at the combined properties of such an implementation:

• A "valid" subset of a character set is not so easy to define. Newline is a control character, and you definitely want to allow that in comments. You'd have work cut out for days or weeks to create a sensible subset of Unicode (and combinations of characters) which could be considered "valid" across the globe.
• An "invalid" subset is also difficult, if you're making anything even remotely complex. For example, you don't want literal quotes in SQL, but you also don't want literal ampersands or inequality signs in HTML, or backslash in JavaScript. If you have a series of input and output languages, the only way to be sure is to escape user input/output for each one, and test that the escaping works.
• The set is valid only for a single version of a single character set, so it's not future-proof.
• You still need to test the full character range to see if there are any security holes.
• If you're not careful with what you accept, you'll end up annoying users, and they will leave. If you're lucky, one in a thousand will file a bug report.

I'd go so far as to say that validating allowed characters reduces security, because it encourages sloppy implementation (lack of testing/escaping). If you escape where necessary you can instead just test the "nasty" characters, and if they work, you have pretty much guaranteed that other nasty characters will also be harmless to the system.

All this is of course not to say that some characters are nonsensical in some fields, such as two in a numeric field. But even this is often not trivial:

• Different languages have different decimal and thousand separators. 1,000 == 1 in much of Europe, and 1'000 is a valid way to write 1000 in some places. You don't want to tell those users their way of writing is wrong.
• Leading zeros, plus signs, hash and asterisk are all valid characters in a phone number. Some countries include an inland prefix ((0) in Switzerland) which you have to use within the country, and have to exclude when using a language code.
• Contrary to many sloppy web developers, email addresses can contain dash and plus characters, and a whole lot of others which are routinely rejected as if to prove that the company policy is that they would do just fine if it wasn't for all those pesky customers.
• Names can contain quote characters (Gerard 't Hooft), numbers (John Doe the 5th), punctuation (John Doe, M.D.), and SQL:

Answer 2

To me it depends on what the condition is.

If only certain characters are valid (passwords (dubious?), phone numbers, hostnames, ip addresses) then you should check that each character is valid.

If however anything is valid but special characters are problems (file names, sql, html) then you should explicitly check for them.

Answer 3

Usually the set of disallowed characters is small, so searching for then in the input stream makes the most sense. For those times when it is reverse -- allowing only a few and disallowing many - it's usually better to match against the accepted characters. There is no universal rule. You have to fully understand what your allowable range of characters are.

You ask whether you're missing something obvious. What you're missing is the perspective of your users. You're approaching the need for validation from a technical perspective, but most validation is all about the user.

Asking if it's important is as easy as asking if it improves the user experience. If it does, you should support it. If it doesn't, don't. User frustration is high when they enter a character only to be told on a separate screen or popup window or error message "sorry, you can't enter a "-" in your phone number". Frustration can be equally high if they know a "-" is legal but your software disallows it.

Most importantly, however, is to be absolutely certain the characters you disallow are genuinely necessary to disallow. For example, nothing frustrates me more than a web site that gives me an error when I put dashes in a phone number or social security number, or spaces in a credit card number. Computers are quite smart enough to simply remove them. Don't force me to input a string that matches your internal requirements when you can easily accept more and simply ignore characters that any reasonable human would know to just ignore.