Writing a language agnostic API?

Question

I'm planning on writing a C++ program which is heavily influenced by plugin modules. Initially I had only thought of making the APi available as an abstract class that others extend. After talking to someone they mentioned that I should think about making it available in a language agnostic way.

How do I effectively expose an API in a manner that any language can use it?

The solution that I could come up with was having the C++ API like I originally intended, for C++ directly and CLI languages (C#, F, etc) and then a sort of document and executable combination. Where expected input is described in the document and then the executable is started (via console/terminal) I would give parameters given as start up commands. This would theoretically work but doesn't sound like a standard solution too me. How is this done in other desktop software?

Answer 1

You cannot do that because some languages implementations could have an ABI different and incompatible with the C one.

On current systems, C is very common and most (but not all) language implementations have an ABI and calling conventions compatible with the one from C. Notice that it is a property of the implementation (your C compiler and your operating system), not of the programming language.

Also, some languages have different and incompatible views on control flow (look into call/cc and tail-calls in Scheme, Goroutines in Go, or backtracking and cut operator in Prolog, CLIPS rules, concurrent actor languages, ....). I'm not sure you'll be able to design something which easily fits all of them (and that could impact the FFI requirements of implementation of these languages).

If using C++, beware of name mangling and of exception handling. (e.g. C longjmp is not friendly with C++ exceptions). Dynamic loading facilities like dlopen and dlsym are relevant to name mangling. So prefer an API using extern "C" functions. libgccjit could be inspirational (it is coded in C++ but has a C friendly API) and perhaps useful (you could consider runtime generation of glue code).

Memory management (notably with garbage collection) is also an issue. Study for examples foreign function interface of Ocaml and of SBCL and of Lua and of Guile. Look into libffi. Some languages also want serialization or persistence facilities (even for foreign data). Read also about dynamic software updating.

You could provide some reflection facilities (e.g an API to query your API, e.g. like GTK introspection). You might try to provide a generic closure mechanism like in GObject-s.

You could use (or customize or adapt) code generators like SWIG. You might consider compiler plugins (e.g. GCC MELT extensions).

If you have a wide and complex API (hundreds of public functions or data types) consider at least providing a machine readable form of it, e.g. some database or XML or JSON from which automatic glue code could be generated.

There is no silver bullet.

^{NB. Better make your stuff free software. You might get outside contributions and you'll need outside help to interface your thing with many programming languages, including some that you don't even know. Open source is a good way to counter leaky abstractions (since you and other contributors can dive into the implementation source code). Given the variety of programming languages, you will need outside help (as soon as your API is successful)}

---

why the question don't make any sense

I asked several times the OP to give much more context and motivation, and to explain the domain for which the library is written, and the context in which that library would be developped. Sadly I've got no answers at all.

The applicative domain (e.g. high speed frequency trading, software for dentists, selling machine, weather forecast, image processing, speech recognition, word processing, social network, static source code analysis ...) matters practically a lot, because it defines what programming languages are likely to be considered and what programming paradigm is probably used. For instance, an image processing library is very unlikely to need to be interfaced to Agda or Coq, but probably would be used from C or C++ applications. Business software is often written in Java (and older ones was in Cobol) so needs to be interfacable to JNI or JVM. Static source code analysis software is very likely to use some theorem prover (so Ocaml or Common Lisp or Haskell is important to them, and they need to be garbage-collection friendly). And so on. Sometimes, inter-process communication (including RPC, MPI, web services, RESTful applications) can be used, but at other times it is not efficient or not convenient enough. Details and context are very important and we don't know them.

The question mentions "desktop software", but that just means today something having a native (non-Web) GUI used with a mouse and a large enough color screen (the dentist software, the software for managing my bank accounts, your favorite game software, the word processor, a web browser are all desktop software, but their main commonality is just having a GUI). Using Qt (a very powerful GUI cross-platform toolkit for C++) facilitates the development of such things. And quite often a library used on desktop (think of XML or JSON libraries like Xerces and JsonCPP, machine learning libraries like TensorFlow, HTTP client libraries like libcurl, numerical computation libraries like BLAS or GMPlib, etc...) don't even care about GUI (but it could care about response time -having functions returning in less than 0.4 seconds- and thread friendliness), because the application (not a reusable library) would care about GUI itself.

The question mentions plugins, but these are simply compiled software components which get dynamically loaded at runtime (thus increasing the virtual address space of the process loading them), using e.g. dlopen on POSIX (and probably LoadLibrary on Windows, which I don't know at all). Notice that the JVM don't use (Java-coded) plugins but dynamically loaded classes, and speaking of plugins don't make sense with JavaScript or with Common Lisp (or most languages having a compile or eval primitive). So the mention of plugins don't means much (both Firefox browser and GCC compiler accept plugins, for very different reasons).

Without a lot more additional details (including applicative domain, context, motivation ...) the current question don't make any sense. I offer to delete my answer if that helps in closing the question in its current form.

^{PS. I don't know and never used Windows (but I use Unix since 1987, and Linux since 1993), but that is an unimportant detail.}

Answer 2

The accepted answer is more or less correct, but doesn't offer you anywhere to go if language independence is actually a burning issue for you. C++ is fairly callable from other languages (C#, Python, etc), and from the tone of your question it sounds more like a "nice to have", but let's pretend otherwise for a second.

If platform/language independent communication is important to you, you should be using a service-oriented architecture. You should provide your C++ code as its own self contained service, which can respond to requests over technology agnostic network protocols.

This is pretty common in the world of server applications and has a lot of benefits to do with architecture, scalability and protecting your applications from nasty memory bugs (+ stack overflows, etc) in other people's code.

It's a little bit more unusual for desktop apps though. I have seen it with the Rider IDE, where they mainly use it for the architectural benefits of keeping their ReSharper product separate from their IDE product.

This is ultimately how you make things agnostic, but also represents a serious architectural change that has a lot of advantages and disadvantageous in multiple areas. Look into it and decide.

Answer 3

Although none is literally 100% universal, there are a number of ways to make the API easily usable from a fairly wide variety of languages (more than most people usually care about, anyway).

The most obvious (and ubiquitous) form would be RPC, such as gRPC or Thrift (or, if you're using Windows, COM).

These all work reasonably similarly: you define the API itself in an interface definition language (IDL). You then use a compiler to generate code for the client and/or server side of this interface. This will typically give you the skeleton of a server. You fill the actual functionality, compile it all up, and off you go.

Answer 4

While I agree with the other answers in principle, I think the OP is expecting at least some starting points which can help OP's further search for an answer.

In-process

• Important consideration

  • Native code is architecture-specific. This means 32-bit and 64-bit can't mix in the same Windows process.
  • While pure .NET IL can be executed in either 32-bit or 64-bit environments, native code must be compiled individually for each architecture.
  • If these limitations are show-stoppers, you must use out-of-process approach instead.

• Win32 DLL

  • Export C-callable functions.
  • Can pass integer values, floating point values, arrays of primitives (including byte arrays), etc.
  • Be careful with "structs", make sure two pieces of communicating software agree on struct field alignment for each struct.
  • .NET users can use P/Invoke to call DLLs. This can be converted into a .NET wrapper, which can completely hide the native aspect of the underlying implementation.
• Microsoft COM
  • Mature, recommended
• .NET Interop using C++/CLI
  • Recommended if most users are on .NET

Out-of-process, same machine, non-communicating (no information exchange in the middle of execution)

• Command line application.
  • Communication happens via command-line arguments and files.
  • Using I/O redirection on Windows is not recommended because it is unreliable except for very simple ones. The documentation mentions deadlocks as a possibility.

Out-of-process, same machine, communicating

• Choice of running as Windows Application or as Windows Service
• Command line application plus any choice of inter-process communication. Refer to inter-process communication on Windows documentation.
  • Shared memory or inter-process memory-mapped file
  • Inter-process synchronization primitives (only for synchronization or semaphore needs)
  • Inter-process pipes
  • Over the file system. Synchronization (hand-off) must happen via one of the other means. Alternatively, repeatedly scan a folder for file changes at periodic intervals.
• WCF

Out-of-process, different machines, communicating

• Architectures
  • Server client architecture
  • Distributed architecture
• Writing an web service that runs under IIS
• Running as a Windows Service that communicates over TCP/IP
• Running as a Windows Service that communicates over WCF

Answer 5

To bring an "old-fashioned", but quite mature technology into the discussion:

CORBA allows you to define object-oriented APIs in a language-independent way (defines its own IDL = interface definition language). Language mappings are available for really all major languages (and even many exotic ones).

It supports local and remote inter-process communication (that's the typical use case), but server implementations can also exist in-process (then using direct method calls). The inter-process communication protocol is binary-based and quite fast.

At our company, we're still using it for performance-critical cases, and interoperability always worked (using mixtures of Java, C, C++, C# and CommonLisp). We've used CORBA for interfaces to partner companies, often not even knowing what programming language they used.

And maybe you've already used it without knowing. E.g. Java's RMI/IIOP is effectively CORBA.