I have a scripting engine I just published as an open source project. It's been sitting on my harddrive waiting for about a year. My engine of course isn't complete in any way, but it does work for simple scripts. It has a javascript-ish feel to it, but I don't wish to abide by the ECMA spec or anything.

Now, the big thing I'm working on is improving code quality while leaving the language working as it is(which I have a few regression tests to "prove"). It doesn't have a formal grammar at all and works like so:

  1. Preprocess/Tokenize. At this point it removes whitespace and cuts everything into "tokens", which is basically just a structure containing a string and a rough "hint" as to what the token is (Number, Identifier, Operation, etc) and some debugging info such as line number
  2. A ScriptingEngine class which takes the list of tokens and actually parses them and executes them
  3. An "ExpressionEvaluator" class which will take a subset of the tokens list and build a specific tree of operations, values, and then execute operations and such and collapse the tree down into a single value

My engine has the goals of being portable(works everywhere .Net does) and self-contained. So far, this "works", but the code is terrible and I'm pretty sure that I'm going about it the wrong way.

I'm wondering if a formal grammar and everything that goes with it might help

Some benefits I've heard of being more formal with grammar

  • Unambiguous specification of the language
  • Easier to maintain/change
  • More traditional/Bigger community support?

And some of the disadvantages

  • Some languages can be very difficult to reduce to a formal grammar, ie Perl.
  • A learning curve for someone not in the know(ie, me)
  • Generally rely on tools such as yacc and ANTLR, which introduce another step in your workflow and/or add dependencies(which I'd like to avoid)

Although this project is in .Net, it could equally apply to any other implementing language. Should I use a formal grammar? Can someone expand on the pros/cons of both sides?

  • 1
    Perl is an example of why you should have a grammar. The real counterexample is perhaps Tcl, which uses a syntactic approach much like you're doing; its formal grammar is keyword-free. (Of course, the real question is “Why are you making another language? What's your unique hook?”) Mar 25, 2013 at 8:49
  • A formal grammar might help you reveal syntactic ambiguities and pitfals (precedences, etc.) on the other hand you have to keep it in sync with your lexer/parser.
    – wirrbel
    Mar 25, 2013 at 8:58
  • 1
    @DonalFellows My unique hook is "because I want to" :)
    – Earlz
    Mar 25, 2013 at 14:18

1 Answer 1


First of all, a formal grammar doesn't imply you have to use a parser generator. You can write a recursive descent parser or whatever floats your boat.

Formal grammars are good as documentation. They help programmers decide what foreign code means and tells you what is allowed / mandatory in the language.

For example, in javascript, do I always have to wrap a function literal in parens if I want to call it right away? That is, is this allowed:

function(a1,a2) { ... }(42, "bacon").

Formal grammars are also very helpful when you want to

  1. Make another implementation of the language,
  2. Do code generation targeting the language,
  3. Or for doing static analysis of the language.

Of course, if it's a quick and dirty scripting language these points are less important.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.