0

This may be a stupid question, and it would certainly take one Hell of a lexer, but do any extant programming languages allow you to do something like:

c# (1.2) {
    // c# code
}
Perl (5) {
    # perl code
    c# (2.0) {
        // some more c# code
    }
}

To utilize multiple languages in individual source files? I'm working on a design for a new language and want to allow something like this in it (basically out of sheer laziness, not wanting to port snippets copied and pasted from the web, older projects, or just for shits and giggles.)

In my research thus far it seems like hacking gcc's intermediary language will be a good way to get it into optimized machine code with the least amount of effort, while the lexer is going to be a challenge (I've been playing around with an IntelliJ plugin for the language [again, laziness, as I'd prefer not to have to craft a whole IDE to put together a mockup of the editor for the language,] but have already hit a hard limit on the context-free nature of the Backus-Naur form of grammar notation, so it seems like this will have to take another format simply for the features I want in my own language without regard to others - making most of the JetBrains tutorials on the matter mostly useless as a result.) This in turn might be a bit of a blessing since it's a chance to abstract away the language definition to a degree with a sort of grammatical bootloader in the lexer and language specs included along the way - e.g. for the above pseudocode something like:

#require c#-1.2
#require Perl-5
#require c#-2.0

At the top of a given source file or somewhere within it's include chain (with my new language as the default and the grammar to build new grammar and parsing methods built into it to bootload the whole process lexically.

Anyway, back to the point: does anything like this already exist that I might use as a touchstone? Especially in relation to dynamical parsers and lexers (where the already-parsed code can change the interpretation of future code significantly - akin to redefining the whole language for a given context [block of code.])

  • 2
    I am assuming you are familiar with Katahdin? – Jörg W Mittag May 30 '18 at 14:42
  • Minor nit pick, but from a Computer Science perspective, whatever your language processor accepts, it's one language, no matter how closely certain parts of it may resemble certain other languages. – Solomon Slow May 30 '18 at 14:50
  • @JörgWMittag Katahdin is new to me, I'm reading up on it now. As far as semantics - they're complex (the language is a mix of JavaScript, c++, c#, Lucid, OpenSCAD and a few others with largely c-like overtones.) Note about that link: most of the spec isn't on that page, the class definition is probably the most up to date. I started the mockup years ago in node.js just to hack it together quickly (a couple years) but am now finishing up the design and adding that lexical bracket notation. – CoryG May 30 '18 at 14:58
  • @JörgWMittag Katahdin is pretty close from the look of it, but I definitely want to partition the languages out in addition to allowing inline modifications to the language (e.g. to be able to target a block of code to an ancient version of a language the compiler happens to know.) – CoryG May 30 '18 at 15:00
  • What kind of interactions to you allow between the sections of code? Should C# be able to call functions in the Perl part and vice versa? – JacquesB May 30 '18 at 15:01
3

Not really. Language interop is a difficult problem, and language embedding even more so.

Many languages have nontrivial syntax constructs that cannot be easily parsed by a general purpose parser. Perl 5 is the prime example of that: in order to parse a Perl 5 program, you need to execute it. E.g. here is a Perl 5 program that will complain about an extra closing brace 50% of the time:

BEGIN {
  print "Running arbitrary code during parsing :)\n";
  *f = rand() > 0.5 ? sub () {} : sub ($) {};
}
for ("the lulz") {
  f /7 } # /
}

How does this work? We define a function f with a prototype. The prototype affects parsing of arguments. The ($) prototype parses one argument, which will here be the result of applying the regex /7 } # / against $_. But the () prototype does not parse any arguments, so f /7 is a division, and # / is a comment.

Consider also that some languages like the Lisp family have a completely different concept of “syntax” than C-like curly brace languages. Indentation-sensitive languages like Python or Haskell have no representation in BNF. So syntactic embedding is really difficult. Instead, embedding will typically use some kind of string or here-doc syntax.

The more difficult problem is semantic interoperability. This is fundamentally not a solvable problem, because languages are insanely different. There are simple considerations like memory management. If I pass a garbage-collected object into a language with manual memory management, how will it be disposed? The object might now be referenced from code in both languages, and the garbage collector may be unable to see some references.

Object systems are another area of great differences. In JavaScript and many other dynamic languages, I can patch methods at runtime. How can this work in C++ where method calls may be resolved at compile time? In Java, object initialization order is effectively parent-before-child. How can existing Java classes work when part of a multiple inheritance design in Python? C doesn't even have a first class object system.

There are also a gazillion little type system differences. One language has immutable strings, another doesn't. In C or C++, you cannot call certain functions unless you have a non-const object. Support for variadic functions is inconsistent, varying between “the only kind of function” (Perl) over “special syntax for an array argument” (Java) to “compile-time template expansion” (C++). Some languages have some kind of namespaces, others don't. Some languages support value semantics for user defined types. Generics differ wildly. Overloaded functions. And so on.

The result of these incompatibilities is that there must be some interface layer. Quite often, this must be extended for user-defined types. Either the objects are converted to a different representation, or they are represented by some kind of wrapper object in the target language. Calling an untyped, variadic function from a static language often ends up being quite uncomfortable. It will never feel native.

There are specific examples where some kind of interop does work well. For example, autogenerating language bindings from one source definition like SWIG. Some language platforms like CLR and JVM define a common semantic model that enables easy interoperability for all languages targeting that platform. There are also some approaches to interoperability that are not invisible, but still work well. For example FFIs, or the Component Object Model. But these generally come with severe semantic restrictions.

I would therefore urge you to abandon your quest for perfect language interoperability between a wide range of languages. Instead, you might have more luck improving interop between two specific languages, e.g. “R in Python” or somesuch. In general, it will be much more difficult to get two languages to interface when they each define their own VM, and easier if the languages can communicate over a simple interface like the C calling convention.

In practice, language interop is so horrendously difficult that in-process interop is often avoided altogether (unless they share a common platform, or one of the languages is C). Instead, inter-process communication over a socket or pipe turns out to be a lot easier. See also: Unix philosophy, microservices.

  • I appreciate your perspective and the detail on this, but the language I'm designing takes these points into account already and it will be executing in a custom managed environment. The question is more about whether or not similar systems exist for a specific facet of it I'm having issues with (specifically, I'm trying to decide between precompiled and dynamically linked parsers for each language and version thereof or inline definitions and purely grammar-based specs, so I'm trying to work out a format for the language specifications as I'd much prefer the latter of the two options.) – CoryG May 30 '18 at 20:21
  • 1
    @CoryG I do hope you succeed because such an environment would be awesome, but I really don't think it is possible. If you do build something, I'd appreciate a link :) – amon May 30 '18 at 20:35
  • 1
    Regarding combining parsers: a single grammar for all languages isn't going to work. Extending each language grammar with syntax to switch to a different parser might work (like Lisp's reader macros). A meta-language that contains segments of target languages (similar to the cweb preprocessor) will work well, but doesn't provide the kind of embedding you're after. – amon May 30 '18 at 20:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.