Should I use a formal grammar for my interpreted scripting language

Question

I have a scripting engine I just published as an open source project. It's been sitting on my harddrive waiting for about a year. My engine of course isn't complete in any way, but it does work for simple scripts. It has a javascript-ish feel to it, but I don't wish to abide by the ECMA spec or anything.

Now, the big thing I'm working on is improving code quality while leaving the language working as it is(which I have a few regression tests to "prove"). It doesn't have a formal grammar at all and works like so:

1. Preprocess/Tokenize. At this point it removes whitespace and cuts everything into "tokens", which is basically just a structure containing a string and a rough "hint" as to what the token is (Number, Identifier, Operation, etc) and some debugging info such as line number
2. A ScriptingEngine class which takes the list of tokens and actually parses them and executes them
3. An "ExpressionEvaluator" class which will take a subset of the tokens list and build a specific tree of operations, values, and then execute operations and such and collapse the tree down into a single value

My engine has the goals of being portable(works everywhere .Net does) and self-contained. So far, this "works", but the code is terrible and I'm pretty sure that I'm going about it the wrong way.

I'm wondering if a formal grammar and everything that goes with it might help

Some benefits I've heard of being more formal with grammar

• Unambiguous specification of the language
• Easier to maintain/change
• More traditional/Bigger community support?

And some of the disadvantages

• Some languages can be very difficult to reduce to a formal grammar, ie Perl.
• A learning curve for someone not in the know(ie, me)
• Generally rely on tools such as yacc and ANTLR, which introduce another step in your workflow and/or add dependencies(which I'd like to avoid)

Although this project is in .Net, it could equally apply to any other implementing language. Should I use a formal grammar? Can someone expand on the pros/cons of both sides?

Answer 1

First of all, a formal grammar doesn't imply you have to use a parser generator. You can write a recursive descent parser or whatever floats your boat.

Formal grammars are good as documentation. They help programmers decide what foreign code means and tells you what is allowed / mandatory in the language.

For example, in javascript, do I always have to wrap a function literal in parens if I want to call it right away? That is, is this allowed:

function(a1,a2) { ... }(42, "bacon").

Formal grammars are also very helpful when you want to

1. Make another implementation of the language,
2. Do code generation targeting the language,
3. Or for doing static analysis of the language.

Of course, if it's a quick and dirty scripting language these points are less important.