I need to extract very particular data from log files(of different types and formats). Since I am a recent college passout; my mind ran to using Lex and Yacc for the purpose. Now I have the following Questions

1. Will it be legal to do so ? (This product I am working for belongs to one of the biggest tech companies in the world.)

2. Also ; I would like to know if I am being too afraid to write my own parser?

3. How can I use Lex and Yacc if my product is Windows based?

Please tell me if you need any clarification or extra information.

  • 3
    With regard to (2), there's not enough information here to know. "Parsing" covers a lot of ground. I wouldn't want to write a C++ parser by myself, but most of the log file parsing I've done was using regular expressions in Perl. Do you need a real parser, or just regexes? – David Thornley May 19 '11 at 14:07
  • Get the dragon book and go to town. Or, you know, just write a few regexp. – Rein Henrichs May 19 '11 at 16:46
  • Or, perhaps: "I need to develop a parser, so I will use Lex and Yacc." Now you have two problems. – Rein Henrichs May 19 '11 at 17:09
  • @david Thornley i need to extract info from logs. you advise please what all i'll need .. – Chani May 19 '11 at 18:34
  • @heinrich i did not understand your second commment – Chani May 19 '11 at 18:34

Before I answer your questions let me first start to say that there are two major classes of computing languages:

  • Regular languages: Most log files follow this format. Each line is a full expression, and can be parsed separately. Essentially, all your needs can be met by just a lexer or regular expressions.
  • Syntactical languages: There is a specific grammar and rules surrounding what makes proper syntax (or sentence structure). Most programming languages and even some markup languages follow this format. Essentially you need both a lexer and a parser.

If your need is parsing log files, you can probably get away with reading them line by line and using a regular expression to get the fields. If the log line is very simple, you can easily hand roll your parser and avoid the syntax ugliness of regular expressions. I've done both approaches and it's fairly quick work.

If you need something a bit more robust, then you can use ANTLR, Flex, Yacc, etc. depending on your platform needs. Understand that ANTLR, while itself is a Java tool, can generate C/C++ and C# code as well as Java code.

Now for your questions one by one:

  1. Is it legal? Absolutely. Most parser generators explicitly put in their licenses that the license does not apply to the generated code. If the source code is proprietary, so is the generated code that came from it.
  2. Am I just being afraid of writing my own parser? Only you can answer that. However, common sense says that if there is a tool to help you save time and avoid errors, you should probably use it. All the more if it doesn't cost extra money. Just make sure the tool you pick fits the job. If all you need to parse is a regular language, then don't opt for the full fledged parser generators.
  3. Yacc and Lex on Windows? You can use an alternative like ANTLR, or you can use the Cygwin based tools, or the Bumble-Bee port. You have options.

There are much easier to learn and more powerful tools available today.

  1. ANTLR this is one of the easiest to use and best supported tools on the market. Great books on how to use it by the author as well.
  2. PyParsing is useful for lighter weight tasks and prototyping things you can then go and re-implement in ANTLR.
  1. Yes, it is. Using GPL'd software places no obligation on you, except that if you extend GPL'd code and distribute the resulting program, you must also distribute the extended source code. So for internal use the GPL is irrelevant. Also, if you don't extend flex or yacc, but just use them, you are under no obligations either (but be aware that the question "What constitutes a derived work?" is controversial.)
  2. Probably. It's not that hard, although parsing log files is usually simple enough that regular expressions are even simpler and faster.
  3. Why wouldn't you? Get the Cygwin versions, or a dedicated Windows port (like e.g. Bumble-Bee), and you're good. The few calls you need can easily be integrated into your build whether you use Make/Ant or Eclipse/Visual Studio/whatever IDE.
  • 2
    In regards to your point #1, the GCC, F/Lex, Yacc libraries all have an additional clause that states that the GPL does not apply to the data (source code) generated by the tool. In short, it is not considered a derivative work. – Berin Loritsch May 19 '11 at 14:53
  • 1
    @Berin Loritsch: To be picky, it isn't the data. Bison etc. create parsing tables, but they also come with a lot of pre-written code that uses those tables. The tables aren't a derivative work, but the software Bison packs into the parser is. That's what the additional clause is about. – David Thornley May 19 '11 at 17:03
  • Thanks for the clarification. The takeaway is that using the parser generator doesn't add any license restrictions to your code. – Berin Loritsch May 19 '11 at 17:07

I think if you use a parser generator, that would be a reasonable and practical way to build a parser. Doing so has its advantages as it's easier to work with a BNF grammar than a lot of code.

That said, it's also at times very educational to write your own parser. One can begin to appreciate lex and yacc and the science behind it once one tackles parsing challenges directly.

Writing your own parser can also have other benefits. First, it can mesh with your existing code base better (if you design it that way). Second, you might see better performance with your own hand-coded tokenizer if it's written pragmatically (rather than, say, regexing every token).


Learn Perl. Perl is the super deluxe version of the Swiss Armyknife of text file wrangling, and by having a few regular expressions with parentesis marking what you need, it is very frequently possible to do even advanced log file extraction in a few lines of Perl.

  • thanks. i have been looking into perl for the last two hours .. – Chani May 19 '11 at 19:25
  • @RYUZAKI: I like Perl, and it's likely the best of the popular languages for parsing log files. I haven't done much Perl on Windows, but rumor is that, for Windows, Strawberry Perl is better than ActivePerl. – David Thornley May 19 '11 at 19:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.