How to compile to multiple target languages

Question

Sidenote, just learned about Haxe which may offer some insight into this question.

The basis of my question is what sorts of tradeoffs (if any) need to or must be made if you wanted to compile one language into several others. I am in particular thinking like this in terms of plugins, but you could also apply it to cross-language compilers.

Basically, for compilers, the goal is to write some code in say JavaScript or perhaps TypeScript, and then compile it to different native platforms/languages/environments like in C, Java, Swift, etc. A middle ground seems to be with plugins for say WordPress, Magento, Shopify, etc.. You might have a simplified template language you are using, or may opt for PHP or some other language as your "primary" language, and then while not compiling to different languages, you are compiling to different lets say "pages" or "template" styles in each platform. On the more extreme end that still seems related is how like with the Fog library in Ruby, they abstract away AWS, Google Cloud, Azure, etc. into a single higher-level abstraction, and you can target any one of these services as a backend. Maybe somewhat similar to that is Homebrew, but not too sure about that.

What I'm mainly wondering is if it's possible to write a generic abstraction layer and have it compile to multiple different targets in these cases and others like it. I know for example Linux abstracts away different architectures so you don't need to write your code specific to an architecture most of the time (all of the time?), but I don't know of a "Higher-Order Operating System Layer" that abstracts all operating systems like Linux, Mac, and Windows, into one thing, so that you can use any of them without regard to the underlying implementation. (Why didn't they do that?)

But basically, can you write a compiler that takes a source algorithm and transforms it into a target language algorithm in a generic way? Say for example, graphics rendering, or http servers. With graphics rendering I know of Metal on the Mac/iOS, WebGL in the browser, the DOM in the browser too, OpenGL on Linux, DirectX on Windows perhaps (I'm reaching, don't know for sure). I can sort of imagine having a generic "graphics" layer that has the concept of sprites and pixels and colors and whatnot, and then it compiling to each of these target platforms in an optimized way. But is this a fantasy? Must you write the algorithm for each particular platform individually? That is, say you write a physics engine in some hypothetical "Higher-Order Graphics Layer". Can it then compile your higher-order physics engine algorithms and data structures down into several totally different target platforms/languages?

What tradeoffs must be made? What limitations are there? Is it even possible? If so, what measures must you take to implement something like that?

For an HTTP server, let's say you just have a simple algorithm like "open socket, loop to listen for connections, etc.". Well, in JavaScript you don't even need to do this, you can just do httpServer.listen(). But in C you might need to allocate some memory and do some stuff in addition to opening sockets and such. So in this case it seems hard (but I don't know, is it impossible or practically infeasible?) to write a generic algorithm for an HTTP server in both C and JavaScript. Is it though? Is there a way to -- I don't know, using some Gang of Four design patterns or something like that -- to write an algorithm for an HTTP server that worked on both platforms (and others)? Or for the game example, same thing?

In the extreme you would just say make game engine and it would make the best game engine for each platform. But short of that, let's say you write a game engine in some higher-order way. Is it even remotely possible to make it compile to several target platforms without rewriting the whole thing from scratch for each platform?

Basically all of these examples hint at the main question: is it even possible to build a transformation of one system into several others in a programming sense? I don't even know if this is an area of research which might have research papers which could help better understand the problems in this area. But either way, I'd like to know if it's possible, and what the tradeoffs are or what you generally have to look out for.

LLVM seems related, but the low-levelness and well-definedness of assembly seems to make the problem a lot easier to solve. I'm not sure if it's possible in the situations I've described here (and others).

Answer 1

I don't know of a "Higher-Order Operating System Layer" that abstracts all operating systems like Linux, Mac, and Windows, into one thing, so that you can use any of them without regard to the underlying implementation.

This sounds exactly like what platforms like the Java Virtual Machine and Microsoft CLR do.

Why didn't they do that?

I'd say they did do that. However, such platforms are not the be all-end all of software development. They present an abstraction of the services need by most programs and provided by most operating systems. However, abstractions always gloss over differences and details. That's why they're abstractions. Sometimes those details are important for the sake of efficiency or security.

For example, you can certainly write a highly functional, fast web server in Java. It will be good enough for many purposes. However, there may be a need for a web server that is not merely "good enough", but which is the fastest possible web server on any platform extant today. You'll have to look at the minute details of each platforms network stack, and you may find that one OS will expose some parameter of the network stack/scheduler that is not readily available on the other platforms, or not as efficiently implemented.

But short of that, let's say you write a game engine in some higher-order way. Is it even remotely possible to make it compile to several target platforms without rewriting the whole thing from scratch for each platform?

You don't typically have to rewrite the whole thing from scratch. More typically you have some sort of facility for conditional compilation. That is, when the base facility differs dramatically on two platforms, you do something like

 #ifdef OS1
 void drawSquare() {
    plainOldDrawSquare()
 }
 #elseif OS2
 void drawSquare() {
   fiddleWithLowLevelGraphicsConfiguration();
   superfastPlatformSpecificDrawSquare()
 }
 #endif

The lines starting with # are directives to the compiler telling it which version of the source code to build for a specific platform.

Answer 2

I know at least one "living" software product which is exactly developed this way: a generic high level language (in this case "AlgoPascal") is compiled to several different other high level languages (C++, C#, Delphi, CPython, there was also a VBA version in the past). So yes, this is definitely possible

Of course, for this case of a scientific mathmatics library, almost no operating system specific libraries or APIs are required. And that is the tradeoff one has to make here: one can only support the smallest common denominator of all target environments - or one has to create a target runtime infrastructure which looks common on all supported operating system platforms (which is the .NET and Java approach: provide a huge common set of libraries which abstract most of the operating system specifics away).

Let add a sidenote on ALGLIB, which is not directly related to your question: for me, it seems the vendor found a way to Open-Source the library this way without disclosing the "real" AlgoPascal source code. Not sure about the legal implications, since the published source code is most probably not what the GPL calls the "preferred form of the work for making modifications" , maybe this is worth a question on Opensource.SE on its own.

Answer 3

It is possible to unify this to a certain degree with a virtual machine as Charles Grant already pointed out. However, there is the issue with the law of leaky abstraction. Whatever you abstract away will leak through at some point. As long as you are just processing abstract data this is no problem. Every VM system is able to map concepts like strings, integers, arrays so some platform specific code.

The difficulty starts when you reach the boundaries of a system. How far can you abstract a way a file system? For example, what about deleting a file that is currently opened by a process? No issue in Linux but a big issue in windows. What about GUIs with all their widget libraries? What about input devices? An interface that requires the ability to right-click works fine if you have a mouse but really bad on a touch-screen.

Therefore all the abstractions of your operating system have to be limited to the common set of functionality for the platforms you are targeting. This is why you will sooner or later run into issues with OS specific code even if you use a language like Java that promises to overcome this problem.