Readable regular expressions without losing their power?

Question

Many programmers know the joy of whipping up a quick regular expression, these days often with help of some web service, or more traditionally at interactive prompt, or perhaps writing a small script which has the regular expression under development, and a collection of test cases. In either case the process is iterative and fairly quick: keep hacking at the cryptic-looking string until it matches and captures what you want and will reject what you don't want.

For a simple case result might be something like this, as a Java regexp:

Pattern re = Pattern.compile(
  "^\\s*(?:(?:([\\d]+)\\s*:\\s*)?(?:([\\d]+)\\s*:\\s*))?([\\d]+)(?:\\s*[.,]\\s*([0-9]+))?\\s*$"
);

Many programmers also know the pain of needing to edit a regular expression, or just code around a regular expression in a legacy code base. With a bit editing to split it up, above regexp is still very easy to comprehend for anyone reasonably familiar with regexps, and a regexp veteran should see right away what it does (answer at the end of the post, in case someone wants the exercise of figuring it out themselves).

However, things don't need to get much more complex for a regexp to become truly write-only thing, and even with diligent documentation (which everybody of course does for all complex regexps they write...), modifying the regexps becomes a daunting task. It can be a very dangerous task too, if regexp is not carefully unit tested (but everybody of course has comprehensive unit tests for all their complex regexps, both positive and negative...).

So, long story short, is there a write-read solution/alternative for regular expressions without losing their power? How would the above regexp look like with an alternative approach? Any language is fine, though a multi-language solution would be best, to the degree regexps are multi-language.

---

And then, what the earlier regexp does is this: parse a string of numbers in format 1:2:3.4, capturing each number, where spaces are allowed and only 3 is required.

Answer 1

A number of people have mentioned composing from smaller parts, but no one's provided an example yet, so here's mine:

string number = "(\\d+)";
string unit = "(?:" + number + "\\s*:\\s*)";
string optionalDecimal = "(?:\\s*[.,]\\s*" + number + ")?";

Pattern re = Pattern.compile(
  "^\\s*(?:" + unit + "?" + unit + ")?" + number + optionalDecimal + "\\s*$"
);

Not the most readable, but I feel like it's clearer than the original.

Also, C# has the @ operator which can be prepended to a string in order to indicate that it is to be taken literally (no escape characters), so number would be @"([\d]+)";

Answer 2

The key to documenting the regular expression is documenting it. Far too often people toss in what appears to be line noise and leave it at that.

Within perl a single /x tells the regular expression parser to ignore most whitespace that is neither backslashed nor within a bracketed character class.

The above regular expression would then become:

$re = qr/
  ^\s*
  (?:
    (?:       
      ([\d]+)\s*:\s*
    )?
    (?:
      ([\d]+)\s*:\s*
    )
  )?
  ([\d]+)
  (?:
    \s*[.,]\s*([\d]+)
  )?
  \s*$
/x;

Yes, its a bit consuming of vertical whitespace, though one could shorten it up without sacrificing too much readability.

And then, what the earlier regexp does is this: parse a string of numbers in format 1:2:3.4, capturing each number, where spaces are allowed and only 3 is required.

Looking at this regular expression one can see how it works (and doesn't work). In this case, this regex will match the string 1.

Similar approaches can be taken in other language. The python re.VERBOSE option works there.

Perl6 (the above example was for perl5) takes this further with the concept of rules which leads to even more powerful structures than the PCRE (it provides access to other grammars (context free and context sensitive) than just regular and extended regular ones).

In Java (where this example draws from), one can use string concatenation to form the regex.

Pattern re = Pattern.compile(
  "^\\s*"+
  "(?:"+
    "(?:"+
      "([\\d]+)\\s*:\\s*"+  // Capture group #1
    ")?"+
    "(?:"+
      "([\\d]+)\\s*:\\s*"+  // Capture group #2
    ")"+
  ")?"+ // First groups match 0 or 1 times
  "([\\d]+)"+ // Capture group #3
  "(?:\\s*[.,]\\s*([0-9]+))?"+ // Capture group #4 (0 or 1 times)
  "\\s*$"
);

Admittedly, this creates many more " in the string possibly leading to some confusion there, can be more easily read (especially with syntax highlighting on most IDEs) and documented.

The key is recognizing the power and "write once" nature that regular expressions often fall into. Writing the code to defensively avoid this so that the regular expression remains clear and understandable is key. We format Java code for clarity - regular expressions are no different when the language gives you the option to do so.

Answer 3

The "verbose" mode offered by some languages and libraries is one of the answers to these concerns. In this mode, whitespace in regexp string is stripped out (so you need to use \s) and comments are possible. Here's a short example in Python which supports this by default:

email_regex = re.compile(r"""
    ([\w\.\+]+) # username (captured)
    @
    \w+         # minimal viable domain part
    (?:\.w+)    # rest of the domain, after first dot
""", re.VERBOSE)

In any language that doesn't, implementing a translator from verbose to "normal" mode should be a simple task. If you're concerned about your regexps' readability, you would probably justify this time investment pretty easily.

Answer 4

Every language that uses regexes allows you to compose them from simpler blocks to make reading easier, and with anything more complicated than (or as complicated as) your example, you should definitely take advantage of that option. The particular trouble with Java and many other languages is that they don't treat regular expressions as "first-class" citizens, instead requiring them to sneak into the language via string literals. This means many quotation marks and backslashes that aren't actually part of regex syntax and make things hard to read, and it also means that you can't get much more readable than that without effectively defining your own mini-language and interpreter.

The prototypical better way of integrating regular expressions was of course Perl, with its whitespace option and regex-quoting operators. Perl 6 extends the concept of building up regexes from parts to actual recursive grammars, which is so much better to use it's really no comparison at all. The language may have missed the boat of timeliness, but its regex support was The Good Stuff(tm).

Answer 5

I like to use Expresso: http://www.ultrapico.com/Expresso.htm

This free application has the following features that I find useful over time:

• You can simply copy and paste your regex and the application will parse it for you
• Once your regex is written, you can test it directly from the application (the application will give you the list of captures, replacements...)
• Once you have tested it, it will generate the C# code to implement it (note that the code will contain the explanations about your regex).

For example, with the regex you just submitted, it would would look like:

Of course, giving it a try is worth a thousand word describing it. Please also note that I'm note related in any way with the editor of this application.

Answer 6

This is an old question and I didn't see any mention of Verbal Expressions so I thought I'd add that information here as well for future seekers. Verbal Expressions were specifically designed to make regex human understandable, without needing to learn the symbol meaning of regex. See the following example. I think this does best what you are asking for.

// Create an example of how to test for correctly formed URLs
var tester = VerEx()
    .startOfLine()
    .then('http')
    .maybe('s')
    .then('://')
    .maybe('www.')
    .anythingBut(' ')
    .endOfLine();

// Create an example URL
var testMe = 'https://www.google.com';

// Use RegExp object's native test() function
if (tester.test(testMe)) {
    alert('We have a correct URL '); // This output will fire}
} else {
    alert('The URL is incorrect');
}

console.log(tester); // Outputs the actual expression used: /^(http)(s)?(\:\/\/)(www\.)?([^\ ]*)$/

This example is for javascript, you can find this library now for many of the programming languages.

Answer 7

For some things, it might help to just use a grammar like BNF. These can be much easier to read than regular expressions. A tool such as GoldParser Builder can then convert the grammar into a parser that does the heavy lifting for you.

The BNF, EBNF, etc. grammars can be much easier to read and make than a complicated regular expression. GOLD is one tool for such things.

The c2 wiki link below has a list of possible alternatives which can be googled, with some discussion on them included. It is basically a "see also" link to top off my grammar engine recommendation:

Alternatives To Regular Expressions

Taking "alternative" to mean "semantically equivalent facility with different syntax", there are at least these alternatives to/with RegularExpressions:

• Basic regular expressions
• "Extended" regular expressions
• Perl-compatible regular expressions
• ... and many other variants...
• SNOBOL-style RE syntax (SnobolLanguage, IconLanguage)
• SRE syntax (RE's as EssExpressions)
• different FSM syntaces
• Finite-state intersection grammars (quite expressive)
• ParsingExpressionGrammars, as in OMetaLanguage and LuaLanguage (http://www.inf.puc-rio.br/~roberto/lpeg/lpeg.html)
• The parse mode of RebolLanguage
• ProbabilityBasedParsing...

Answer 8

The simplest way would be to still use regex but build your expression from composing simpler expresssions with descriptive names e.g. http://www.martinfowler.com/bliki/ComposedRegex.html (and yes this is from string concat)

however as an alternative you could also use a parser combinator library e.g. http://jparsec.codehaus.org/ which will give you a full recursive decent parser. again the real power here comes from composition (this time functional composition).

Answer 9

I like how Raku has redesigned the regular expression syntax.

It is so much cleaner, extensible, and frankly more powerful.

Here is a direct translation of your regex into a Raku regular expression.

/
    :sigspace

    ^

    [
        [(\d+) ':']?
        [(\d+) ':']
    ]?

    (\d+)

    [
        <[,.]> (\d+)
    ]?

    $
/;

The most noticeable thing is of course the whitespace.
You don't realize how nice adding a few spaces can be until you are prevented from doing so.

---

Perhaps the second most noticeable in the above translation is :sigspace.

The thing that really makes it noticeable is that I didn't have to write a single \s* even though it works as if I had.

What :sigspace does is translate whitespace in the regular expression into allowing whitespace in the string to be matched.
(Normally whitespace in a regex does nothing.)

Below is a short example of its use.

The first line uses :sigspace.
The second line is the direct translation of what it compiles into.
The third line is the traditional way you would write it.

' 123 456 ' ~~ /:s ^ (\d+) (\d+) $/

' 123 456 ' ~~ /^ <.ws> (\d+) <.ws> (\d+) <.ws> $/

' 123 456 ' ~~ /^  \s*  (\d+)  \s+  (\d+)  \s*  $/

The <ws> token is smart. It knows if whitespace should be optional or mandatory.
Between (\d+) and (\d+) it is mandatory because otherwise you wouldn't be able to tell where one ends and the other starts.
(The . in <.ws> is there to indicate it isn't important enough to be included in the match result object.)

---

The next thing is that (?:…) has been renamed to […].

In fact every (?SOMETHING … ) is gone.

(?i:  … )  →  [ :i  … ]  or  [ :ignorecase  … ]
(?-i: … )  →  [ :!i … ]  or  [ :!ignorecase … ]

(?<= … )  →  <?after … >
(?<! … )  →  <!after … >

(?=  … )  →  <?before … >
(?!  … )  →  <!before … >

(?> \w+ )  →  \w+:
           →  # see also `:ratchet` mode

(?(condition)yes-regexp|no-regexp)
  →  [ <?{ condition }> yes-regexp || no-regexp ]

(?<named> …)  →  $<named> = …

Note that something like <foo> is a function call.
And <?foo> is a function call where only the truthiness of the result matters.
So then <!foo> must be a function call where only the untruthiness of the result matters.

The (?(condition)yes-regexp|no-regexp) wasn't really replaced with anything. The replacement code is just the combination of features that happen to work together to make adding such a feature uneccesary.

:ignorecase is an adverb that turns on ignorecase mode.
Adverbs are something that are everywhere in Raku.
(It doesn't need to be in [] unless you need to constrain it to a small part of your regex.)

---

Something which you probably won't notice until it is pointed out, is that regular expressions are code. It is just code with a different default syntax and semantics.

'AAABBBCCC' ~~ /
                   ^

                   $<a> = ('A'+:)

                   {}
                   :my $count = $<a>.chars; # <-- this is a regular variable

                   $<b> = ('B' ** {$count})
                   $<c> = ('C' ** {$count})

                   $
               /

Since it is just code, you can embed or call out to regular Raku code to add or extend any feature you may want.
What that means is all of the weird extensions to regexs that other languages have added are unnecessary.
(They may get added with a nice syntax if they are useful enough.)

---

For more complex regular expressions, you can create a grammar.

grammar Foo {
  rule TOP {^
    <colon-part> <point-part>?
  }
  rule colon-part {
    <num> ** 1..3 % ':'
    # <num> repeated 1..3 times, each separated by ':'
  }
  rule point-part {
    <[.,]> <num>
  }
  token num { \d+ }
}

say Foo.parse( '1:2:3.4' );

｢1:2:3.4｣
 colon-part => ｢1:2:3｣
  num => ｢1｣
  num => ｢2｣
  num => ｢3｣
 point-part => ｢.4｣
  num => ｢4｣

Which is roughly the same structure as:

{
  'colon-part' => {
    'num' => [ '1', '2', '3' ],
  }
  'point-part' => {
    'num' => '4',
  }
}

The syntax is such that you could create that same multi-level match data structure using a single regex.

/:s ^
    $<colon-part> = (
        [$<num> = (\d+)] ** 1..3 % ':'
    )
    $<point-part> = (
        <[.,]> $<num> = (\d+)
    )?
    $
/

Note that (…) creates a new scope, while […] doesn't.

You can also combine segments.

my $num = /\d+/;
my $colon-part = /:s <num=$num> ** 1..3 % ':' /;
my $point-part = /:s <[.,]> <num=$num> /;
'1:2:3.4' ~~ /:s ^ <colon-part=$colon-part> <point-part=$point-part>? $/

The reason for <num=$num> is so that the result of the regex in $num gets stored under the name of num in the result object. Otherwise it could have just been <$num>.

You could of course just put the parts of the grammar into the lexical namespace.

my token num {\d+}
my rule colon-part { <num> ** 1..3 % ':' }
my rule point-part {:s <[.,]> <num> }
'1:2:3.4' ~~ /:s ^ <colon-part> <point-part>? $/

Answer 10

I thought it'd be worth mentioning logstash's grok expressions. Grok builds upon the idea of composing long parsing expressions from shorter ones. It allows convenient testing of these building blocks and comes prepackaged with over 100 commonly used patterns. Other than these patterns, it allows use of all regular expressions syntax.

The above pattern expressed in grok is (I tested in the debugger app but could have blundered):

"(( *%{NUMBER:a} *:)? *%{NUMBER:b} *:)? *%{NUMBER:c} *(. *%{NUMBER:d} *)?"

The optional parts and spaces make it seem a bit uglier than usual, but both here and in other cases, using grok can make one's life much nicer.

Answer 11

In F# you have the FsVerbalExpressions module. It allows you to compose Regexes from verbal expressions, it also has some pre-built regexes (like URL).

One of the examples for this syntax is the following:

let groupName =  "GroupNumber"

VerbEx()
|> add "COD"
|> beginCaptureNamed groupName
|> any "0-9"
|> repeatPrevious 3
|> endCapture
|> then' "END"
|> capture "COD123END" groupName
|> printfn "%s"

// 123

If you're not familiar with F# syntax, groupName is the string "GroupNumber".

Then they create a Verbal Expression (VerbEx) which they construct as "COD(?<GroupNumber>[0-9]{3})END". Which they then test on the string "COD123END", where they get the named capture group "GroupNumber". This results in 123.

I honestly find the normal regex much easier to comprehend.

Answer 12

First, understand that code that merely works is bad code. Good code also needs to accurately report any errors encountered.

For example, if you're writing a function to transfer cash from one user's account to another user's account; you wouldn't just return a "worked or failed" boolean because that doesn't give the caller any idea of what went wrong and doesn't allow the caller to inform the user properly. Instead, you might have a set of error codes (or a set of exceptions): couldn't find destination account, insufficient funds in source account, permission denied, can't connect to database, too much load (retry later), etc.

Now think about your "parse a string of numbers in format 1:2:3.4" example. All the regex does is report a "pass/fail" that doesn't allow adequate feedback to be presented to the user (whether this feedback is an error message in a log, or an interactive GUI where the errors are shown in red as the user types, or whatever else). What types of errors does it fail to describe properly? Bad character in first number, first number too large, missing colon after first number, etc.

To convert "bad code that merely works" into "good code that provides adequately descriptive errors" you have to break the regex up into many smaller regexes (typically, regexes that are so small that it's easier to do it without regexes in the first place).

Making the code readable/maintainable is just an accidental consequence of making the code good.