Would Rebol (or Red) benefit from Lisp-style Macros?

Question

As a 'seasoned' Rebol developer with some knowledge of the world outside, I'd be curious as to the utility/pitfalls of implementing Lisp-style macros in Rebol (and/or Red).

My understanding (always happy to revise) is that Lisp is able to preprocess code prior to evaluation/compilation, modifying that code according to the rules of the applicable macros. This can allow for more complex/expressive statements than otherwise, with efficiencies as those statements are expanded only once as a body of code is loaded. The macro rules themselves are relatively straightforward as the homoiconicity of Lisp allows the code to be manipulated by the same terms one would modify data.

As Rebol is also a homoiconic language, it stands that it too could accommodate macros between loading and evaluation with similarly expressive transformation grammar (though I'd posit that Rebol's more freeform evaluation model—statements not generally bound by parentheses—makes it harder to identify the portions of code to be transformed).

My own inclination is that in the main, macros aren't really necessary in Rebol—you can already create complex code structures that are fairly efficient, and where more efficiency is desired, Rebol has the capabilities to accommodate without significant loss of expressivity. On the other had, 'fairly' and 'no significant loss' could be considered weasel words and macros do offer an efficiency/expressivity bump, but at what implementation cost?

Also, as in Rebol you can load code prior to evaluation, it would be possible to implement a mechanism for load-expand-evaluate at the user-level permitting third-party macro implementations.

This post is partly a response to this question and everything (pretty much) I know about macros comes from this exchange.

Update

Entering the World of Macros

Red introduced a rudimentary macro system in version 0.6.2. I have used them effectively to create cross-compatible scripts that target both the Ren-C branch of Rebol 3 and Red.

Answer 1

This is the first time I hear about Rebol, but from a quick look at the Wikipedia page it seems to me that Rebol dialects are just like Lisp macros: they both receive ordinary code of the language that has passed lexical and syntactic(but not semantic!) processing, and process it with their own semantical rules.

So, Rebol will not benefit from Lisp macros, because it already has that feature under a different name, and having two core-language features that do exactly the same thing is bad - especially for languages that rely on Homoiconicity, where uniform simplicity in the syntax is the key to their magic.

Update

The asker explainer in the comments that macros have the advantage that they are expanded once, before the code execution, and injected into the code.

This, IMO, will make things worse, because it would mess up the order of parsing!

For example, let's say we have the Rebol command foo x bar y, where foo is a dialect and bar is a macro. What should the interpreter do? Depends on foo's arit:

• If foo accepts a single argument, that means we have two commands: foo x and bar y. So bar y needs to be expanded.
• If foo accepts three arguments, that means that we have one command, and bar is the second argument of foo. This means bar shouldn't be expanded, and instead passed to foo as-is.

The rules are clear - for a human, and probably for static compilers as well. But Rebol is dynamic and interpreted, so foo is only evaluated when it needs to be executed - but bar should be expanded before that happens! So now the interpreted needs to evaluate the dialects before it expands the macros, so it can expand the macros before it evaluated the dialect, and round and round the interpreted goes trying to resolve the circle...

Lisp doesn't have this problem, because:

• It's easy to know where a function/macro begins, because you need to put them in parenthesis. You can't have foo x bar y - you need to either have (foo x bar y) or (foo x) (bar y) or even (foo x (bar y)), so it's easy for the interpreter to know what you are trying to do.
• Lisp doesn't have dialects - only functions and macros. And since functions don't control the evaluation of their arguments - macros always evaluate themselves. If Rebol didn't have dialects, and foo was a function, foo x bar y would require the interpreter to first evaluate x, bar and y, and since bar is a macro it would have expanded regardless of foo's arity, because foo has no control whatsoever on the evaluation of it's arguments.

Answer 2

Most Probably Not.

Lisp Macros appear to have two big purposes. First is the DRY ("don't-repeat-yourself") aspect, enabling the capture of patterns that would otherwise be difficult-or-impossible in the language. Second is a performance advantage that comes from being able to apply arbitrary computation at compile-time.

A good number of cases where Lisp programmers appear to require macros for expression is because ordinary Lisp functions are limited in how they can work with "context-dependent code fragments" passed as arguments. Rebol attacks this problem by making bindings of individual symbols "travel along" with their code fragments. The result is that many cases requiring macros in Lisp can be written as ordinary Rebol functions.

Furthermore, Rebol hinges its "DRY" story on dialects. Dialects are essentially the idea of processing "quoted" code literally under new rules, distinct from the core evaluator. This competes with macros as a strategy for language extension...and it's also incompatible. Lisp itself does not run macros in quoted portions of code, and there would not be a general way to do parameterizations and expansions in an unknown dialect's "grammar".

The performance angle only affects Red (as Rebol is unlikely to ever be compiled). But if compile-time services are available to optimize function-shaped-things, then the only benefit macros could offer here would be for things that could not be shaped like functions. Truly motivating cases would probably find better expression as a dialect, which would presumably also be able to take advantage of compile-time services.

(Expanded upon in Rebol vs. Lisp Macros)

Answer 3

Rebol probably would not, since there is no explicit compilation step. An extra 'macro expansion' pass would have to be added to implement Lisp macros properly, which is not all that different from the expand-* suggestion by @rgchris.

Lisp macros "work" because everything which enters a Lisp image is technically required to go through a compilation step by the Common Lisp specification. That pass is what determines the result of (eval-when) calls, figures out how to handle control flow bits and pieces, do JIT-ting, and general purpose code magery. Lisp code goes through the reader, yet there is always an existing image there to call back to. In a sense, the Lisp runtime and compiler are working in tandem. As opposed to C, where the compiler is working by itself. Rebol is similar to C in that aspect; the interpreter is running by itself. You need both for Lisp macros to make sense.

Since Red has a planned JIT and carries along Red/System which is both JIT or compiled ahead of time, the ability to immitate Lisp macros would be advantageous. Most of the work done to support macros is already done by necessity by both JIT and AOT compilers, as they already must scan functions and parameter signatures to determine what kind of assembly code will be produced. All that remains at that point is to decide whether a statement is part of a statement to compile, or a statement to run in the compiler's Red context and treat the return value as a statement to compile. Since running eval is a frowned-upon practice in general, this means the only time a macro could potentially change the state of your Red environment is during the bootstrapping or development phases of your program--you already modify the environment in those, so you expect it.

Lisp also has the (macroexpand-1) and (macroexpand-all) facilities, which allow you to see what the Lisp environment is going to use as the expansion of a given macro. This is surprisingly more useful for debuggability than normal template systems--you can ask for an exact copy of the template expansion, which is not something you can ask of any C++ compiler. I mention this because someone ultimately always brings up how macro systems can make debugging slightly more difficult; yet, a system which is even more difficult to debug is being used on a much more popular language on a daily basis.

An example of this usefulness is that PARSE from Red could be used to create a dialect which accepts the definition of a binary file format, which then outputs a block of Red/System code, which would then ultimately be compiled. Since execution is only necessary at compile time the entire code which does the parsing, as well as the specification text itself, are not needed in the binaries. The entire preprocessor could also be removed, since everything from #if to #define and #import would be achievable using a Lisp-like macro system.