Making a language agnostic Plugin system

Question

I want to make some software which relies heavily on Plugins. I plan on writing it in C++ and will most likely be getting plugins built in C++, Python and possibly Java But most likely C++ and python but since I don't want to rule out any languages that may be slightly more obscure or uncommon. How do I design a plugin that I can use in my C++ program that is unbound to language (or allows for the broadest compatibility)?

One way that I have thought that this could be done is to offer an API in C++ (as that allows for native code to work) and design a system that will call executable files with parameters being passed as arguments. As long as my program and the Plugin have a specification of what will be passed in and what should happen with the output this should (theoretically) be a valid way of getting maximum compatibility.

The plugins will be responsible for processing data in the form of file(s) and output more files. The actual task that the plugin completes is dependent on the Plugin but the output should always be a file. The plugin will run in parallel to my program with it being called when there is a file to process. There could be multiple instances of a single plugin running at the same time, working of different files. There is no limit to the amount of time that it takes to complete the task (from my programs point of view), but the faster it is done the better. Plugins do not affect the main program directly, only when being given files to work on and collecting results, which is why I think my executable strategy may work for my specific use case.

Answer 1

How do I design a plugin that I can use in my C++ program that is unbound to language (or allows for the broadest compatibility)?

You cannot do that for exactly the same reasons explained in my answer to your previous question on Writing a language agnostic API? (which is a duplicate of this one).

You need to tell more about your context, your motivation your domain. You are seeking for a holy grail design which does not (and probably cannot) exist.

If universal designs existed, you'll already read about them.

Software design is always a matter of trade-off and compromises for a particular problem in a particular domain.

As I told in my other answer, plugins for GCC (a free software compiler, you can study all its source code) are different in their design and goals than plugins for Firefox (a free software web browser).

BTW both GCC and Clang/LLVM are free software compilers for C++ accepting plugins, and even if these two software are similar in function, their plugin design is very different and incompatible. The zsh and fish shells also accept plugins, and so does Python or Guile or Lua interpreters, or the emacs or vim editors.

The plugins will be responsible for processing data in the form of file(s) and output more files. The actual task that the plugin completes is dependent on the Plugin but the output should always be a file.

There are many languages which have very different views on file IO. Look for example into the IO monad of Haskell (which I don't know well). It is very different of IO in Ocaml or in C++.

The plugin will run in parallel to my program with it being called when there is a file to process.

I guess that your usage of "plugin" is wrong.

^{You should try to find a better word, or give your (unconventional) definition of plugins, or at least illustrate that word by existing examples.}

A plug-in generally involves some dynamically loaded (using dlopen on POSIX, Load Library on Windows) code (which extends the virtual address space on a process). On Linux, every program written in C++ or C claiming to accept plugins (e.g. gcc, firefox, vim, gedit, vlc, xmms2, Qt and GTK applications, Apache or Lighttpd web servers, ....) is using dlopen(3) (perhaps indirectly) to load the plugin at runtime, and all the loaded plugins are technically shared objects in ELF.

^{(I don't know Windows, but I guess that all plugins loaded by C++ programs on Windows are DLLs, and they are loaded with LoadLibrary or similar function)}

---

You might not even need any plugin, and you could instead consider inter-process communication.

You might be thinking of software pipelines (but they are not plugins). Then think more in terms of communication protocols. Read about named pipes and anonymous pipes. Look (perhaps for inspiration) into the old unix pipelines (you might want to mimic them on Windows, and some C++ frameworks such as POCO or Qt could be helpful). Read about message passing.

A software pipeline (and notably unix pipelines) enable to make various programs written in different programming languages work together. But you do need to specify some communication protocol. Old Unix utilities supposed a line-by-line protocol, but you might define your messages to be something very different (e.g. JSON, see JSON/RPC for inspiration).

You could also make your C++ application extensible by embedding some interpreter (like Lua, Guile, or even Python ...) or some bytecode VM (like Parrot, NekoVM, ...) in it. But you should not speak of plugins in that case! You probably should speak of extensions, not plugins. For an extremely well known -and decades old- example, Emacs is an extensible editor (I'm using it daily since the 1990s). You can improve its behavior by customizing it in extension written in E-Lisp. Another example (less familiar to me): AutoCAD is a proprietary program extensible in AutoLISP; MicroSoft Word is rumored to be extensible thru macros coded in some Visual Basic dialect. A common way to make a software extensible is indeed to embed some interpreter in it. Of course such an approach is not language agnostic, it is tied to the scripting language accepted by that interpreter.

So my concrete recommendation is to embed some good enough interpreter (like Lua or Guile) in your product, and accept the fact that such extensions won't be language agnostic. In practice, if you choose a good enough extension language, it does not matter much. Both Lua and Guile have been designed as extension languages (and so was Tcl, but I don't recommend using it). My personal preference is Guile (because Scheme is a very elegant and powerful language, quite small and easy to learn, read SICP).

The plugins will be responsible for processing data in the form of file(s) and output more files

If it is only for that (and nothing more ...), you don't need any plugin. You could simply decide that if an output file path name starts with e.g. an exclamation point ! or a vertical bar | your program would pipe to some external command (BTW such conventions are already used in several Unix programs), or have some program argument conventions to behave like that. On POSIX systems, you'll just for example decide to use popen(3) instead of fopen(3), and that external command could be implemented in any language. You then have a primitive form of software pipelines (which you should of course not call a plugin).

However, I still believe that extending your program by embedding an interpreter is preferable.

Answer 2

Process-boundary plugins are a great fit where the compromises you have to make for them (discussed below) are acceptable.

I work on a project with many different types of plugins and for our application, the compromises don't cause any problems. There are many ways to interface with them; the design choice for ours was that the plugin programs communicate with JSON across the standard I/O. Since putting it into production earlier this year, we've been very happy with the decision because it's made the process of expanding our system easier for us as developers and for our users, too.

Pros:

• Developer Flexibility. Plugins can be written in any language the author finds comfortable or to be the right tool for the job.
• Evolvability. Your system's technology stack can be changed without having to rewrite all of the plugins at the same time.
• Safety. If a plugin fails in a catastrophic way, your program is close enough to hear the kaboom but far enough away that it won't be hit by any of the shrapnel.
• Testability. The plugin can be tested and tinkered with at the command line without having to link them with your system or into complex test jigs.

Cons, both having to do with overhead:

• Serialization. Data you pass to another program has to be in some standard format that probably won't be the same as it exists in memory. This means that there will be time spent encoding and decoding data at both ends of the transaction. You'll also have to pick a format that has libraries available for as wide a variety of languages as plugin authors are likely to use.

• Startup. If you're doing high-volume calls to programs that are short-lived and have high startup overhead (Python is the poster child for the latter), you'll find the load on your system goes up considerably. We ran into this on my project and switched one of the plugins from single-use to streaming, where the program is started once and given one blob of work at a time to chew on. (This was accompanied by code to maintain variably-sized pools so programs that went unused didn't linger too long.) For your application, this doesn't sound like a problem, but for others, this plus serialization and communication may add unacceptable latency.