What is meant by "Now you have two problems"?

Question

There is a popular quote by Jamie Zawinski:

Some people, when confronted with a problem, think "I know, I'll use regular expressions." Now they have two problems.

How is this quote supposed to be understood?

Answer 1

Some programming technologies are not generally well-understood by programmers (regular expressions, floating point, Perl, AWK, IoC... and others).

These can be amazingly powerful tools for solving the right set of problems. Regular expressions in particular are very useful for matching regular languages. And there is the crux of the problem: few people know how to describe a regular language (it's part of computer science theory / linguistics that uses funny symbols - you can read about it at Chomsky hierarchy).

When dealing with these things, if you use them wrong it is unlikely that you've actually solved your original problem. Using a regular expression to match HTML (a far too common occurrence) will mean that you will miss edge cases. And now, you've still got the original problem that you didn't solve, and another subtle bug floating around that has been introduced by using the wrong solution.

This is not to say that regular expressions shouldn't be used, but rather that one should work to understand what the set of problems they can solve and can't solve and use them judiciously.

The key to maintaining software is writing maintainable code. Using regular expressions can be counter to that goal. When working with regular expressions, you've written a mini computer (specifically a non-deterministic finite state automaton) in a special domain specific language. It's easy to write the 'Hello world' equivalent in this language and gain rudimentary confidence in it, but going further needs to be tempered with the understanding of the regular language to avoid writing additional bugs that can be very hard to identify and fix (because they aren't part of the program that the regular expression is in).

So now you've got a new problem; you chose the tool of the regular expression to solve it (when it is inappropriate), and you've got two bugs now, both of which are harder to find, because they're hidden in another layer of abstraction.

Answer 2

Regular expressions - particularly non trivial ones - are potentially difficult to code, understand and maintain. You only have to look at the number of questions on Stack Overflow tagged [regex] where the questioner has assumed that the answer to their problem is a regex and have subsequently got stuck. In a lot of cases the problem can (and perhaps should) be solved a different way.

This means that, if you decide to use a regex you now have two problems:

1. The original problem you wanted to solve.
2. The support of a regex.

Basically, I think he means you should only use a regex if there's no other way of solving your problem. Another solution is probably going to be easier to code, maintain and support. It may be slower or less efficient, but if that's not critical ease of maintenance and support should be the overriding concern.

Answer 3

It's mostly a tongue-in-cheek joke, albeit with a grain of truth.

There are some tasks for which regular expressions are an excellent fit. I once replaced 500 lines of manually written recursive descent parser code with one regular expression that took around 10 minutes to fully debug. People say regexes are hard to understand and debug, but appropriately-applied ones are not nearly as hard to debug as a huge hand-designed parser. In my example, it took two weeks to debug all the edge cases of the non-regex solution.

However, to paraphrase Uncle Ben:

With great expressivity comes great responsibility.

In other words, regexes add expressivity to your language, but that puts more responsibility on the programmer to choose the most readable mode of expression for a given task.

Some things initially look like a good task for regular expressions, but aren't. For example, anything with nested tokens, like HTML. Sometimes people use a regular expression when a simpler method is more clear. For example, string.endsWith("ing") is easier to understand than the equivalent regex. Sometimes people try to cram a large problem into a single regex, where breaking it into pieces is more appropriate. Sometimes people fail to create appropriate abstractions, repeating a regex over and over instead of creating a well-named function to do the same job (perhaps implemented internally with a regex).

For some reason, regexes have a weird tendency to create a blind spot to normal software engineering principles like single responsibility and DRY. That's why even people who love them find them problematic at times.

Answer 4

Jeff Atwood brings out a different interpretation in a blog post discussing this very quote: Regular Expressions: Now You Have Two Problems (thanks to Euphoric for the link)

Analyzing the full text of Jamie's posts in the original 1997 thread, we find the following:

Perl's nature encourages the use of regular expressions almost to the exclusion of all other techniques; they are far and away the most "obvious" (at least, to people who don't know any better) way to get from point A to point B.

The first quote is too glib to be taken seriously. But this, I completely agree with. Here's the point Jamie was trying to make: not that regular expressions are evil, per se, but that overuse of regular expressions is evil.

Even if you do fully understand regular expressions, you run into The Golden Hammer problem, trying to solve a problem with regular expressions, when it would have been easier and more clear to do the same thing with regular code (see also CodingHorror: Regex use vs. Regex abuse).

There is another blog post which looks at the context of the quote, and goes into more detail than Atwood: Jeffrey Friedl's Blog: Source of the famous “Now you have two problems” quote

Answer 5

There are a few things going on with this quote.

1. The quote is a restatement of an earlier joke:

   Whenever faced with a problem, some people say "Lets use AWK." Now, they have two problems. — D. Tilbrook

   It is a joke and a real dig, but it's also a way of highlighting regex as a bad solution by linking it with other bad solutions. It's a great ha ha only serious moment.

2. To me—mind you, this quote is purposely open to interpretation—the meaning is straight forward. Simply announcing the idea of using a regular expression has not solved the problem. In addition, you've increased the cognitive complexity of the code by adding an additional language with rules that stand apart from whatever language you are using.

3. Although funny as a joke, you need to compare the complexity of a non-regex solution with the complexity of the regex solution + the additional complexity of including regexes. It may be worthwhile to solve a problem with a regex, despite the additional cost of adding regexes.

Answer 6

RegularExpressionsarenoworsetoreadormaintainthananyotherunformattedcontent;indeedaregexisprobablyeasiertoreadthanthispieceoftexthere-butunfortunatelytheyhaveabadreputationbecausesomeimplementationsdon'tallowformattingandpeopleingeneraldon'tknowthatyoucandoit.

_{^{(Regular Expressions are no worse to read or maintain than any other unformatted content; indeed a regex is probably easier to read than this piece of text here - but unfortunately they have a bad reputation because some implementations don't allow formatting and people in general don't know that you can do it.)}}

Here's a trivial example:

^(?:[^,]*+,){21}[^,]*+$

Which isn't really that difficult to read or maintain anyway, but is even easier when it looks like this:

(?x)    # enables comments, so this whole block can be used in a regex.
^       # start of string

(?:     # start non-capturing group
  [^,]*+  # as many non-commas as possible, but none required
  ,       # a comma
)       # end non-capturing group
{21}    # 21 of previous entity (i.e. the group)

[^,]*+  # as many non-commas as possible, but none required

$       # end of string

That's a bit of an over-the-top example (commenting $ is akin to commenting i++) but clearly there should be no problem reading, understanding, and maintaining that.

So long as you're clear as to when regular expressions are suited and when they're a bad idea, there's nothing wrong with them, and most times the JWZ quote doesn't really apply.

Answer 7

In addition to ChrisF's answer - that regular expressions "are difficult to code, understand and maintain", there's worse: they're just powerful enough to trick people into trying to use them to parse things they can't, like HTML. See the numerous questions on SO on "how do I parse HTML?" For instance, the single most epic answer in all of SO!

Answer 8

Regular expressions are very powerful, but they have one small and one big problem; they are hard to write, and near impossible to read.

In a best case the use of the regular expression solves the problem, so then you only have the maintenance problem of the complicated code. If you don't get the regular expression just right, you have both the original problem and the problem with unreadable code that doesn't work.

Sometimes regular expressions are referred to as write-only code. Faced with a regular expression that needs fixing, it's often faster to start from scratch than to try to understand the expression.

Answer 9

The problem is that regex is a complicated beast, and you only solve your problem if you use regex perfectly. If you don't, you end up with 2 problems: your original problem and regex.

You claim that it can do the work of a hundred lines of code, but you could also make the argument that 100 lines of clear, concise code is better than one line of regex.

If you need some proof of this: You can check out this SO Classic or simply comb through the SO Regex Tag

Answer 10

The meaning has two parts:

• First, you didn't solve the original problem.
  This probably refers to the fact that regular expressions often offer incomplete solutions to common problems.
• Second, you now added additional difficulty associated with the solution you've picked.
  In the case of regular expressions, the additional difficulty probably refers to complexity, maintainability, or the additional difficulty associated with making regular expressions fit a problem it wasn't supposed to solve.

Answer 11

As you ask for it in 2014, it would be interesting to focus on programming languages ideologies of 1997 context comparing to today's context. I will not enter this debate here but opinions about Perl and Perl itself have greatly changed.

However, to stay in a 2013 context (de l'eau a coulé sous les ponts depuis), I would suggest to focus on reenactment in quotes using a famous XKCD comic that is a direct quote of Jamie Zawinski's one :

First I had problems to understand this comic because it was a reference to the Zawinski quote, and a quote of a Jay-z song lyrics, and a reference of GNU program --help -z flag2, so, it was too much culture for me to understand it.

I knew it was fun, I was feeling it, but I didn't really know why. People are often doing jokes about Perl and regexes, especially since it's not the hipstiest programming language, don't really know why it is supposed to be fun... Maybe because Perl mongers do silly things.

So the initial quote seems to be a sarcastic joke based on real life problems (pain?) caused by programming with tools that hurts. Just like a hammer can hurt a mason, programming with tools that are not the ones that a developer would choose if he could can hurt (the brain, the feelings). Sometimes, great debates about which tool is the best occurs, but it's almost worthless cause it's a problem of your taste or your programming team taste, cultural or economic reasons. Another excellent XKCD comic about this :

I can understand people feeling pain about regexes, and they do believe that another tool is better suited for what regexes are designed for. As @karl-bielefeldt answers your question with great expressivity comes great responsibility, and regexes are especially concerned by this. If a developer don't care of how s-he deals with regexes, it will eventually be a pain for people who will maintain the code later.

I will finish with this answer about quotes reenactment by a quote showing a typical example from Damian Conw ay's Perl Best Practices (a 2005 book).

He explains that writing a pattern like this:

m{'[^\\']*(?:\\.[^\\']*)*'}

...is no more acceptable than writing a program like this:

sub'x{local$_=pop;sub'_{$_>=$_[0
]?$_[1]:$"}_(1,'*')._(5,'-')._(4
,'*').$/._(6,'|').($_>9?'X':$_>8
?'/':$")._(8,'|').$/._(2,'*')._(
7,'-')._(3,'*').$/}print$/x($=).
x(10)x(++$x/10).x($x%10)while<>;

But it can be rewritten, it's still not pretty, but at least it's now survivable.

# Match a single-quoted string efficiently...
m{ '            # an opening single quote
    [^\\']*     # any non-special chars (i.e., not backslash or single quote)
    (?:         # then all of...`
    \\ .        # any explicitly backslashed char
    [^\\']*     #    followed by any non-special chars
    )*          # ...repeated zero or more times
    '           # a closing single quote
}x

This kind of rectangular shaped code is the second problem not regexes that can be formatted in a clear, maintainable and readable way.

Answer 12

If there is one thing you should learn from computer science, it is Chomsky hierarchy. I would say that all problems with regular expressions come from attempts to parse context-free grammar with it. When you can impose a limit (or think you can impose a limit) to nesting levels in CFG, you get those long and complex regular expressions.

Answer 13

Regular expressions are more suitable for tokenisation than for full-scale parsing.

But, a surprisingly large set of things that programmers need to parse are parseable by a regular language (or, worse, almost parseable by a regular language and if you only write a little more code...).

So if one is habituated to "aha, I need to pick text apart, I'll use a regular expression", it's easy to go down that route, when you need something that's closer to a push-down automaton, a CFG parser or even more powerful grammars. That usually ends in tears.

So, I think the quote isn't so much slamming regexps, they have their use (and well-used, they're very useful indeed), but the over-reliance on regexps (or, specifically, the uncritical choice of them).

Answer 14

jwz is simply off his rocker with that quote. regular expressions are no different than any language feature - easy to screw up, hard to use elegantly, powerful at times, inappropriate at times, often well documented, often useful.

the same could be said for floating point arithmetic, closures, object-orientation, asynchronous I/O, or anything else you can name. if you don't know what you are doing, programming languages can make you sad.

if you think regexes are hard to read, try reading the equivalent parser implementation for consuming the pattern in question. often regexes win because they are more compact than full parsers...and in most languages, they are faster as well.

don't be put off of using regular expressions (or any other language feature) because a self-promoting blogger makes unqualified statements. try things out for yourself and see what works for you.

Answer 15

My favourite, in-depth answer to this is given by the famous Rob Pike in a blog post reproduced from an internal Google code comment: http://commandcenter.blogspot.ch/2011/08/regular-expressions-in-lexing-and.html

The summary is that it's not that they are bad, but they are frequently used for tasks for whcih they are not necessarily suited, especially when it comes to lexing and parsing some input.

Regular expressions are hard to write, hard to write well, and can be expensive relative to other technologies... Lexers, on the other hand, are fairly easy to write correctly (if not as compactly), and very easy to test. Consider finding alphanumeric identifiers. It's not too hard to write the regexp (something like "[a-ZA-Z_][a-ZA-Z_0-9]*"), but really not much harder to write as a simple loop. The performance of the loop, though, will be much higher and will involve much less code under the covers. A regular expression library is a big thing. Using one to parse identifiers is like using a Ferrari to go to the store for milk.

He says a lot more than that, arguing that regular expressions are useful in, e.g. disposable matching of patterns in text editors but should rarely be used in compiled code, and so on. It's worth a read.

Answer 16

This is related to Alan Perlis' epigram #34:

The string is a stark data structure and everywhere it is passed there is much duplication of process. It is a perfect vehicle for hiding information.

So if your choose the character string as your data structure (and, naturally, regex-based code as the algorithms to manipulate it), you have a problem, even if it works: bad design around an inappropriate representation of data which is hard to extend, and inefficient.

However, often it doesn't work: the original problem isn't solved, and so in that case you have two problems.

Answer 17

Regexes are widely used for quick and dirty text parsing. They are a great tool for expressing patterns that are a little bit more complex than just a plain string match.

However as regexes get more complex serveral issues raise their head.

1. The syntax of regexes is optimised for simple matching, most characters match themselves. That is great for simple patterns but once you end up with more than a couple of levels of nesting you end up with something looking more like line noise than well structured code. I guess you could write a regex as a series of concatenated strings with indentation and comments in-between to show the structure of the code but it seems to be rare for that to actually happen.
2. Only certain types of text matching are well suited to regexes. Often you find yourself getting a quick and dirty regex based parser for some kind of markup language working but then you try to cover more corner cases and you find the regexes getting more and more complex and less and less readable
3. The time complexity of a regex may be non-obvoius. It is not that difficult to end up with a pattern that works great when it matches but has O(2^n) complexity under certain cases of non-matching.

Thus it's all too easy to start with a text processing problem, apply regular expressions to it and end up with two problems, the original problem you were trying to solve and dealing with the regular expressions that are attempting to solve (but not solving correctly) the original problem.