0

I'm trying to find the best way to process about 20,000 lines of a statute in text format into HTML so that it can be displayed and manipulated in a granular fashion, i.e. by clause. The format is essentially a nested list like (a)(1)(A)(i)(II)(aa) and is fairly consistent. Each clause begins with a new line and is enclosed in parenthesis like above.

The goal is for each clause to be enclosed in a paragraph tag with a unique id like p id="sec101a1AiI" (sorry I can't figure out the code formatting). The point is to be able to refer to a specific clause later on. I've played with regular expressions but they don't seem to allow me to iterate through the entire statute and tag things in a unique fashion. I'm sure there are better options using a custom script but I don't know where to start looking for that. Any suggestions would be appreciated.

  • 1
    What have you tried with regex? What languages are you working in? How are you currently processing the statutes into html? – user40980 Jan 27 '14 at 23:35
  • one of the possible combinations (perl, markdown): first a good set of regex to replace your current format, and Text::Markdown in Perl to create your HTML (if you really need it). – marcocs Jan 27 '14 at 23:43
  • Trying to build HTML with regex seems almost as bad of an idea to me as trying to parse HTML with regex. It might be possible, depending on how "regular" the headings actually are, but seems way easier just to write a 20-30 line parser for it (which might in fact use regex but not exclusively so). – Aaronaught Jan 28 '14 at 3:28
  • Thanks for the comments. It sounds like a Perl parser might be the way to do what I'm looking to do. Regex is only allowing me to select, for example, an (I) buried underneath (a)[some text](1)[some text], etc. I can then find/replace all those to give those clauses an ID. This doesn't solve the problem of needing to have a unique ID though. For example, there are going to be about 80 paragraph (a)s so the id needs to be specific to the particular section, such as id="101a...". The HTML is basically just paragraph tags, the problem is really how to approach the parsing. – jkmuller Jan 28 '14 at 3:34
2

This is fairly simple to do, depending on how the input is structured. Let's assume we have a file like

Legalese Introduction

(1) Some Legalese Section

    With another paragraph inside it

    (a) With some sub-paragraph

        (I) Some other statutes

        (II) Some other statutes

(2) Another Section

We can easily split this file into paragraphs and then loop through them. For each paragraph with some enumeration, we can check whether this increments the counter of the current numbering level. If so, we perform that operation and emit an ID.

Otherwise, this either adds another level or closes one or more levels.

We do this by modelling the counters with a stack. On each enumerated paragraph, we pop or push stuff onto the stack.

The difficulty is deciding whether two enumerations are of the same numbering scheme, e.g. L and I could be uppercase latin (A, B, C, …) or roman numerals (I, II, III, …). You might therefore want to transform both into their numeric value and assert that they are not only of the same scheme but are also consecutive.

Here is a basic Perl script that would parse the above example, but would have difficulties with more complex input for the mentioned reasons:

use strict;
use warnings;
use HTML::Entities qw< encode_entities >;

local $/ = ''; # paragraph mode
my @stack;
while(my $section = <>) {
    $section =~ s/^\s+//;
    $section =~ s/\s+$//;

    my ($number) = $section =~ m/[(] ([A-Z]+|[a-z]+|[0-9]+|[IVXLCDM]+) [)]/x;
    if (not $number) {
       print "<p>", encode_entities($section), "</p>\n";
       next;
    }

    # check if the numbering is a continuation of the current or another level
    if (@stack and grep { compatible($_, $number) } @stack) {
      pop @stack until compatible($stack[-1], $number);
      $stack[-1] = $number
    }
    # open a new level
    else {
      push @stack, $number;
    }

    my $id = join "-", "section", @stack;
    print qq(<p id="$id">), encode_entities($section), "</p>\n";
}

sub compatible {
   my ($prev, $this) = @_;
   (my $expected = $prev)++; # increment is magic and works on strings too.
   no warnings 'numeric';
   # numbers
   return 1 if not (grep /[^0-9]/, $prev, $this) and $expected == $this;
   # upper latin
   return 1 if not (grep /[^A-Z]/, $prev, $this) and $expected eq $this;
   # lower latin
   return 1 if not (grep /[^a-z]/, $prev, $this) and $expected eq $this;
   # roman numerals
   return 1 if not (grep /[^IVXLCDM]/, $prev, $this); # TODO check consecutive
   return 0;
}
  • This is really helpful, thank you so much for your response. Unfortunately I cannot give you a vote for lack of reputation. – jkmuller Jan 29 '14 at 3:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.