2

I've been looking at ways to let my app's users write plugins for it. However, to give them more options, I decided to implement a polyglot plugin system.

From the engineering perspective there are multiple ways to do this.

  • Transpiling: Not many libraries that can reliably do this.
  • Bindings: Types can be a pain.
  • Using APIs: Seems like the easiest route, but plugins end up being complex.

One method I thought of was to use Jupyter kernels. Sending messages(via IPC) between event loops that already run in the respective interpreters. I only want to support interpreted/dynamic languages(at least 5) so it will be easier to load at run-time.

Are there are any common methods that I have missed out on and how do I go about building an application like this? What do I have to watch out to prevent the most common pitfalls?

All this has to work while still supporting some sort of event loop. However, I don't need threading since python doesn't have good support for it anyways.

  • How does data transfer (e.g. syncing of objects) happen across the two processes (e.g. serialization, or only send modified pieces of data upon request)? Does syncing happens automatically for certain types of objects automatically, or only when manually? Suppose one process has already entered a event handler. If the event handler's code want to get some other data from the other process, is the interprocess design able to handle that, and also to handle multiple level nesting of such interactions? – rwong Feb 17 '18 at 10:32
  • So, both process A and B have their own respective event loops. So the data transfer happens when one loop requests data it does not have. For eg, process B doesn't have a library implemented because of limitations. So it requests that data and waits. Oh and they're both accessing the same db. So a lot of the data is simply available already. Only run-time state changes need to be transferred. – daegontaven Feb 17 '18 at 11:39
2

Cross-language event loops aren't entirely impossible, but if you go that route you are in for a world of pain. This alone excludes some solutions such as autogenerated language bindings.

The only straightforward way to connect any two programming languages is through IPC. Let your application start plugins as subprocesses, and communicate via pipes. Alternatively, implement your application as a server that separate processes can connect to. Added benefit of IPC-based solutions: most event loops make it easy to trigger on readable I/O handles.

Over these pipes, you exchange messages of any format. This could be a simple “one JSON document per line” protocol, Protobuf messages, or full-on HTTP. This is the the part were you can choose a design that is easy to implement for all participants, both in regards to the message serialization technology and to the protocol implemented by these messages.

How the plugin protocol should be structured depends on what these applications are supposed to be doing. E.g. a plugin may be a filter that massages some data. Then, the main application has code like this:

sendMessageToPlugin(data)
result = await readMessageFromPlugin()

So this is effectively synchronous communication with the plugin, but doesn't block your application. The plugin must still respond to messages in order, but that may not be a problem for many workloads.

Another simple case is when the plugin is merely an event consumer. So you just send messages to the plugin without having to receive any responses. But this opens up some difficult questions: What if there are more events than the plugin can currently consume? How can they be buffered? Should they be buffered? What happens when the plugin crashes?

In these more complicated cases, it may be sensible to use an existing message queue that serves as a broker between your plugins and your application.

It is generally sensible to offer a small, simple, well-defined API to plugins. This makes it impossible for the plugin to request arbitrary data. Instead, you define messages that contain the necessary data. If you need a shared datamodel, using an external database may be sensible because it can do atomic updates, and ensures consistency of your datamodel. Your messages would then contain database keys, not the full data.

Note that the further your design evolves in some direction, the easier it gets to apply best practices from SOA or Microservice approaches. All of these address the same problem: language-agnostic cooperation between multiple processes.

I would also like to point out that these engineering challenges, while not unsolvable, do still require lots of effort. Unless you have a concrete need that justifies the cost of developing this plugin system, it may be better to stay within the host language. Integrating a library is so much easier than dealing with IPC.

  • What did you mean when you said integrating a library is more easier? Did you mean i just fork it out into a sub process and then use IPC? – daegontaven Feb 18 '18 at 1:02
  • With “integrating a library” I mean writing the plugins in Python so that you can just import them, without having to deal with IPC or cross-language issues. – amon Feb 18 '18 at 10:51
  • Well the reason I didn't want to do that is because I expect users to be able to write plugins in any of the supported languages. If they don't know python, they can't write a plugin for it. – daegontaven Feb 19 '18 at 0:42
  • @daegontaven Of course. I just wanted to point out that this kind of plugin system is expensive, so there needs to be a strong business case. If a language-agnostic plugin system costs $200k (made-up number) plus increased long-term maintenance complexity, it needs to also bring in at least $200k in sales. A Python-only plugin system might be 10× cheaper, which could then be far more profitable. – amon Feb 19 '18 at 9:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.