Suppose I am writing a C++ library that I intend to distribute in binary form, with interfaces from other languages (e.g. Python). The 'easy' approach of just compiling the library and distributing the DLL or Framework does not work well.

For it to work you need to compile the library with every supported compiler and every supported compiler option, and bad things can happen if you don't.

The problem is because C++'s ABI is in general not stable, and the ABI of the STL is definitely not stable. A sort of solution is to stick to 'simple' C++ in your public API - simple classes with basic types. The problem with that is you don't get to use the STL's nice types like std::string and `std::vector and end up reimplementing them.

So I'm wondering if there is a better solution using a library Interface Definition Language (IDL). There are loads of these for network protocols, like Thrift, Protobuf, gRPC, CapnProto, etc. Is there one for libraries?

The ideal solution would then take this IDL file, generate a C->C++ wrapper around the C++ library, so that its ABI is now the C ABI. It could then also generate open source wrappers around the C library for whatever language you wanted (including C++).

I know it is kind of insane to wrap C++ with a C API and then wrap that with a C++ API. But I can't see a better way.

Does this exist? Is it insane? Is there a better way?

  • 2
    Well, by now there is actually an acceptably stable C++ ABI for all modern desktop platforms and most others. Of course, most compilers support many ways to change the ABI. Whether the parts of the libraries you want to use are stable enough is a different question, but naturally you cannot generally expect to be able to switch two API-compatible libraries and stay ABI-compatible. – Deduplicator Aug 9 '17 at 16:54
  • Apache Thrift (and I'm sure some of the others you've mentioned) are protocol agnostic. They could be used over the network, but they are not limited to that. I would take another look at Thrift because I'm fairly certain with the Thrift compiler you can just generate what you want from the Thriftfile. Thrift does support containers and strings. I'm not sure what it would produce for C++ – Samuel Aug 9 '17 at 20:52
  • 4
    Writing a C library that happens to use a lot of C++ code internally is a sufficiently sane approach. Under your constraint that the lib must stay closed-source, the only real alternative is to stop trying to build a shared library and instead building a self-contained server that accepts requests via some IPC/network protocol. I have a hunch that just writing a bit of C boilerplate is going to be a lot easier. – amon Aug 9 '17 at 20:53

Yes, there are such things for libraries that aren't (necessarily) accessed via a network. Microsoft's COM would be one obvious example. IBM's nigh obsolete OS/2 did much the same (using an implementation of CORBA, if memory serves).

Though it's largely fallen from favor, CORBA insulated most components from the transport layer, so it could act as basically a wrapper around a library on the same system, or a Thrift-like layer for talking to a server on some other system.

The closest (currently popular) thing I can think of on Linux is D-bus, which serves sort of a similar purpose, but is based primarily on messaging rather than function calls. These can serve roughly the same purposes, but from your viewpoint (wrapper around a library) they're still quite a bit different.

Since you brought up Thrift, I'll point out that Thrift uses quite a simple model for its transport layers, so using it as a wrapper around libraries on a single machine is reasonably feasible as well. The most obvious route on Unix-like systems would probably be to use Unix-domain sockets for the communication. This gives a socket interface, but the communication is (at least normally) via kernel-managed shared memory buffers, so despite the network-like interface, it's quite fast.

  • 1
    COM might be old, but I wouldn't consider it obsolete, it is still used for many core Windows API systems. – whatsisname Aug 10 '17 at 4:22
  • @whatsisname: Read more carefully. I said OS/2 was obsolete, not COM. – Jerry Coffin Aug 10 '17 at 4:32
  • Yeah I definitely want to have a proper DLL interface rather than a local network-like interface. I have bad memories of COM. It's also way more heavyweight than what I want. I guess if I want this I'll have to write it myself. – Timmmm Aug 10 '17 at 12:20

To build on @amon's comment, wrapping C++ code with a C header file is what you want to do, and it is widely used and has many benefits.

Why should you reduce C++ libraries external interface to a C header file?

  • Many popular languages specifically have means of calling C functions natively (e.g., C++, Objective C, Fortran, Cython, Golang). Languages which don't allow for native calls to C functions almost all have a standard way of wrapping C code in order to call it (Java, Python, Matlab).
  • The C standard changes very little compared to C++, which greatly reduces the risk of parts of your header file using deprecated features. Many libraries are still distributed with ANSI C header files.
  • Exposing C++ objects in header files generally breaks encapsulation, meaning changes to your library are likely to break downstream code (you can forward declare classes but this leads to some other challenges).
  • Greatly limits the surface area of your library (minimizes coupling of downstream code).
  • No name mangling.

You can easily pass STL's std::vector and std::string into C functions fairly easily using the .data() and .size() functions. This many times means you need two functions calls: 1) one where the library responds how much memory it needs, and 2) where the allocated memory is passed into the library and filled/modified. Which comes to an important point, when writing a library, don't mix memory allocation and de-allocation across the library boundaries (if the library allocates it, the library should free it, and if the caller allocates it, the caller should free it). Passing around objects like std::vector() would violate this, as internally it allocates, reallocates, and deallocates memory.

  • 1
    Yes but the question is about how to do this automatically using an IDL. – Timmmm Aug 12 '17 at 20:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.