Is there an interface definition language for software libraries?

Question

Suppose I am writing a C++ library that I intend to distribute in binary form, with interfaces from other languages (e.g. Python). The 'easy' approach of just compiling the library and distributing the DLL or Framework does not work well.

For it to work you need to compile the library with every supported compiler and every supported compiler option, and bad things can happen if you don't.

The problem is because C++'s ABI is in general not stable, and the ABI of the STL is definitely not stable. A sort of solution is to stick to 'simple' C++ in your public API - simple classes with basic types. The problem with that is you don't get to use the STL's nice types like std::string and `std::vector and end up reimplementing them.

So I'm wondering if there is a better solution using a library Interface Definition Language (IDL). There are loads of these for network protocols, like Thrift, Protobuf, gRPC, CapnProto, etc. Is there one for libraries?

The ideal solution would then take this IDL file, generate a C->C++ wrapper around the C++ library, so that its ABI is now the C ABI. It could then also generate open source wrappers around the C library for whatever language you wanted (including C++).

I know it is kind of insane to wrap C++ with a C API and then wrap that with a C++ API. But I can't see a better way.

Does this exist? Is it insane? Is there a better way?

Answer 1

Yes, there are such things for libraries that aren't (necessarily) accessed via a network. Microsoft's COM would be one obvious example. IBM's nigh obsolete OS/2 did much the same (using an implementation of CORBA, if memory serves).

Though it's largely fallen from favor, CORBA insulated most components from the transport layer, so it could act as basically a wrapper around a library on the same system, or a Thrift-like layer for talking to a server on some other system.

The closest (currently popular) thing I can think of on Linux is D-bus, which serves sort of a similar purpose, but is based primarily on messaging rather than function calls. These can serve roughly the same purposes, but from your viewpoint (wrapper around a library) they're still quite a bit different.

Since you brought up Thrift, I'll point out that Thrift uses quite a simple model for its transport layers, so using it as a wrapper around libraries on a single machine is reasonably feasible as well. The most obvious route on Unix-like systems would probably be to use Unix-domain sockets for the communication. This gives a socket interface, but the communication is (at least normally) via kernel-managed shared memory buffers, so despite the network-like interface, it's quite fast.

Answer 2

To build on @amon's comment, wrapping C++ code with a C header file is what you want to do, and it is widely used and has many benefits.

Why should you reduce C++ libraries external interface to a C header file?

• Many popular languages specifically have means of calling C functions natively (e.g., C++, Objective C, Fortran, Cython, Golang). Languages which don't allow for native calls to C functions almost all have a standard way of wrapping C code in order to call it (Java, Python, Matlab).
• The C standard changes very little compared to C++, which greatly reduces the risk of parts of your header file using deprecated features. Many libraries are still distributed with ANSI C header files.
• Exposing C++ objects in header files generally breaks encapsulation, meaning changes to your library are likely to break downstream code (you can forward declare classes but this leads to some other challenges).
• Greatly limits the surface area of your library (minimizes coupling of downstream code).
• No name mangling.

You can easily pass STL's std::vector and std::string into C functions fairly easily using the .data() and .size() functions. This many times means you need two functions calls: 1) one where the library responds how much memory it needs, and 2) where the allocated memory is passed into the library and filled/modified. Which comes to an important point, when writing a library, don't mix memory allocation and de-allocation across the library boundaries (if the library allocates it, the library should free it, and if the caller allocates it, the caller should free it). Passing around objects like std::vector() would violate this, as internally it allocates, reallocates, and deallocates memory.