2

I want to make some software which relies heavily on Plugins. I plan on writing it in C++ and will most likely be getting plugins built in C++, Python and possibly Java But most likely C++ and python but since I don't want to rule out any languages that may be slightly more obscure or uncommon. How do I design a plugin that I can use in my C++ program that is unbound to language (or allows for the broadest compatibility)?

One way that I have thought that this could be done is to offer an API in C++ (as that allows for native code to work) and design a system that will call executable files with parameters being passed as arguments. As long as my program and the Plugin have a specification of what will be passed in and what should happen with the output this should (theoretically) be a valid way of getting maximum compatibility.

The plugins will be responsible for processing data in the form of file(s) and output more files. The actual task that the plugin completes is dependent on the Plugin but the output should always be a file. The plugin will run in parallel to my program with it being called when there is a file to process. There could be multiple instances of a single plugin running at the same time, working of different files. There is no limit to the amount of time that it takes to complete the task (from my programs point of view), but the faster it is done the better. Plugins do not affect the main program directly, only when being given files to work on and collecting results, which is why I think my executable strategy may work for my specific use case.

  • 4
    Possible duplicate of Writing a language agnostic API? – amon Oct 7 '17 at 16:07
  • 1
    Not really that question is about the API if you read the top second part answer you can see why is this is a different problem to solve – user3797758 Oct 7 '17 at 16:15
  • 1
    This may be hair-splitting, but this question is about a specific implementation of a solution discussed in the other one. (Not that I'm biased or anything after having written an answer. :-)) – Blrfl Oct 7 '17 at 16:38
  • 1
    I'm pretty sure that your usage of plugin is wrong, and you just want a software pipeline. – Basile Starynkevitch Oct 7 '17 at 17:32
  • 2
    It is very sad that most of your questions don't give enough context and motivation. Plese try to improve that in your future questions by telling more about your goals and your work context. – Basile Starynkevitch Oct 8 '17 at 14:38
10

How do I design a plugin that I can use in my C++ program that is unbound to language (or allows for the broadest compatibility)?

You cannot do that for exactly the same reasons explained in my answer to your (poorly asked) previous question on Writing a language agnostic API? (which is a duplicate of this one).

You cannot expect a universal answer or seek for a universal design.

You need, as always (and apparently, you still don't want to...) to tell more about your context, your motivation your domain. You are seeking for a holy grail design which does not (and probably cannot) exist.

If universal designs existed, you'll already read about them.

Software design is always a matter of trade-off and compromises for a particular problem in a particular domain.

(if you think of games, as for some of your older questions, you should tell that in your question)

As I told in my other answer, plugins for GCC (a free software compiler, you can study all its source code) are different in their design and goals than plugins for Firefox (a free software web browser).

BTW both GCC and Clang/LLVM are free software compilers for C++ accepting plugins, and even if these two software are similar in function, their plugin design is very different and incompatible.

The plugins will be responsible for processing data in the form of file(s) and output more files. The actual task that the plugin completes is dependent on the Plugin but the output should always be a file.

There are many languages which have very different views on file IO. Look for example into the IO monad of Haskell (which I don't know well). It is very different of IO in Ocaml or in C++.

The plugin will run in parallel to my program with it being called when there is a file to process.

I guess that your usage of "plugin" is wrong.

You should try to find a better word, or give your (unconventional) definition of plugins, or at least illustrate that word by existing examples.

A plug-in generally involves some dynamically loaded (using dlopen on POSIX, Load Library on Windows) code (which extends the virtual address space on a process). On Linux, every program written in C++ or C claiming to accept plugins (e.g. gcc, firefox, vim, gedit, vlc, xmms2, Qt and GTK applications, Apache or Lighttpd web servers, ....) is using dlopen(3) (perhaps indirectly) to load the plugin at runtime, and all the loaded plugins are technically shared objects in ELF.

(I don't know Windows, but I guess that all plugins loaded by C++ programs on Windows are DLLs, and they are loaded with LoadLibrary or similar function)


You might not even need any plugin, and you could instead consider inter-process communication.

You might be thinking of software pipelines (but they are not plugins). Then think more in terms of communication protocols. Read about named pipes and anonymous pipes. Look (perhaps for inspiration) into the old unix pipelines (you might want to mimic them on Windows, and some C++ frameworks such as POCO or Qt could be helpful). Read about message passing.

A software pipeline (and notably unix pipelines) enable to make various programs written in different programming languages work together. But you do need to specify some communication protocol. Old Unix utilities supposed a line-by-line protocol, but you might define your messages to be something very different (e.g. JSON, see JSON/RPC for inspiration).

You could also make your C++ application extensible by embedding some interpreter (like Lua, Guile, or even Python ...) or some bytecode VM (like Parrot, NekoVM, ...) in it. But you should not speak of plugins in that case! You probably should speak of extensions, not plugins. For an extremely well known -and decades old- example, Emacs is an extensible editor (I'm using it daily since the 1990s). You can improve its behavior by customizing it in extension written in E-Lisp. Another example (less familiar to me): AutoCAD is a proprietary program extensible in AutoLISP; MicroSoft Word is rumored to be extensible thru macros coded in some Visual Basic dialect. A common way to make a software extensible is indeed to embed some interpreter in it. Of course such an approach is not language agnostic, it is tied to the scripting language accepted by that interpreter.

So my concrete recommendation is to embed some good enough interpreter (like Lua or Guile) in your product, and accept the fact that such extensions won't be language agnostic. In practice, if you choose a good enough extension language, it does not matter much. Both Lua and Guile have been designed as extension languages (and so was Tcl, but I don't recommend using it). My personal preference is Guile (because Scheme is a very elegant and powerful language, quite small and easy to learn, read SICP).

The plugins will be responsible for processing data in the form of file(s) and output more files

If it is only for that (and nothing more ...), you don't need any plugin. You could simply decide that if an output file path name starts with e.g. an exclamation point ! or a vertical bar | your program would pipe to some external command (BTW such conventions are already used in several Unix programs), or have some program argument conventions to behave like that. On POSIX systems, you'll just for example decide to use popen(3) instead of fopen(3), and that external command could be implemented in any language. You then have a primitive form of software pipelines (which you should of course not call a plugin).

However, I still believe that extending your program by embedding an interpreter is preferable.

NB: It is really sad you don't give more motivations and contexts in your questions. What kind of program are you working on? In what domain? For what audience and users? Why do you ask? Please try hard to give more motivation and context in your future questions, and don't be shy to edit them for improvement. That would help a lot people trying to answer them (and future readers too).

5

Process-boundary plugins are a great fit where the compromises you have to make for them (discussed below) are acceptable.

I work on a project with many different types of plugins and for our application, the compromises don't cause any problems. There are many ways to interface with them; the design choice for ours was that the plugin programs communicate with JSON across the standard I/O. Since putting it into production earlier this year, we've been very happy with the decision because it's made the process of expanding our system easier for us as developers and for our users, too.

Pros:

  • Developer Flexibility. Plugins can be written in any language the author finds comfortable or to be the right tool for the job.
  • Evolvability. Your system's technology stack can be changed without having to rewrite all of the plugins at the same time.
  • Safety. If a plugin fails in a catastrophic way, your program is close enough to hear the kaboom but far enough away that it won't be hit by any of the shrapnel.
  • Testability. The plugin can be tested and tinkered with at the command line without having to link them with your system or into complex test jigs.

Cons, both having to do with overhead:

  • Serialization. Data you pass to another program has to be in some standard format that probably won't be the same as it exists in memory. This means that there will be time spent encoding and decoding data at both ends of the transaction. You'll also have to pick a format that has libraries available for as wide a variety of languages as plugin authors are likely to use.

  • Startup. If you're doing high-volume calls to programs that are short-lived and have high startup overhead (Python is the poster child for the latter), you'll find the load on your system goes up considerably. We ran into this on my project and switched one of the plugins from single-use to streaming, where the program is started once and given one blob of work at a time to chew on. (This was accompanied by code to maintain variably-sized pools so programs that went unused didn't linger too long.) For your application, this doesn't sound like a problem, but for others, this plus serialization and communication may add unacceptable latency.

  • I might be misunderstanding the question, answer or both so I could be wrong, but how exactly does this answer the question? Maybe this part: "The programs all communicate with JSON across the standard I/O"? The rest is benefits of having such a system, and the overhead. But the overhead depends on the implementation, which, if I understand correctly, is what OP is asking for right? – NickL Oct 8 '17 at 0:30
  • @NickL I took the question to include the second paragraph and answered it in terms of "that's one way to do it, here's how one project did it and these are the trade-offs." I understand your interpretation even if I don't share it. – Blrfl Oct 8 '17 at 12:35
  • Ah now I see, you basically elaborated on some of the pros and cons of the technique OP had in mind. However, most of them really depend on the implementation. For example, you say plugins can be written in any language, but how would you run and communicate with a plugin written in PHP? While I see your point, I was expecting these kinds of details. – NickL Oct 8 '17 at 17:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.