How to design a C++ program to allow for runtime import of functions?

Question

today, I like to ask you a question towards the capabilities of C++ to realize a specific software architecture.

Of course, I have used the search but have not found any directly linked answer.

Basically, my goal is to build a program which allows the user for modeling and simulation of arbitrarily composed physical systems, e.g. a driving car. I assume to have a library of physical models (functions within classes). Each function may have some inputs and return some outputs depending on the underlying physical description, e.g. a combustion engine model, an aerodynamic drag model, a wheel model, etc.

Now, the idea is to provide the user a framework which allows him to compose any functions according to his needs, i.e. to map any physical behavior. The framework should provide functionalities to connect the outputs and inputs of different functions. Therefore, the framework provides a container class. I call it COMPONENT, which is able to hold one or many model objects (FUNCTION). These containers can also hold other components (cf. composite pattern) as well as the connections (CONNECTOR) between the function parameters. Additionally, the component class provides some general numeric functionalities such as math solver and so on.

The composition of functions should be done during runtime. In the first instance, the user should be able to set up a composition through importing an XML which defines the composition structure. Later, one could think of adding a GUI.

To give you a better understanding here is a very simplified example:

<COMPONENT name="Main">
  <COMPONENT name="A">
    <FUNCTION name="A1" path="lib/functionA1" />
  </COMPONENT>
  <COMPONENT name="B">
    <FUNCTION name="B1" path="lib/functionB1" />
    <FUNCTION name="B2" path="lib/functionB2" />
  </COMPONENT>
  <CONNECTIONS>
    <CONNECTOR source="A1" target="B1" />
    <CONNECTOR source="B1" target="B2" />
  </CONNECTIONS>        
</COMPONENT>

It is not necessary to dive deeper into the framework's capabilities because my problem is much more general. When the framework code/program is compiled, the physical problem description, as well as the user-defined functions, are not known. When the user selects (via XML or later via a GUI) a function, the framework should read the function information, i.e. should get the information of input and output parameters, in order to offer the user the option to interconnect the functions.

I know the principles of reflection and I am aware that C++ does not provide this feature. However, I am sure that the concept of "building objects during runtime" is very often required. How should I set up my software architecture in C++ to achieve my goal? Is C++ the right language? What do I overlook?

Thanks in advance!

Cheers, Oliver

Answer 1

In pure standard C++, you cannot "allow for runtime import of functions"; according to the standard, the set of C++ functions is statically known at build-time (in practice, link-time) since fixed from the union of all translation units composing your program.

In practice, most of the time (excluding embedded systems) your C++ program runs above some operating system. Read Operating Systems: Three Easy Pieces for a good overview.

Several modern operating systems allow dynamic loading of plugins. POSIX notably specifies dlopen & dlsym. Windows has something different LoadLibrary (and an inferior linking model; you need to explicitly annotate the functions concerned, provided, or used by plugins). BTW on Linux you can practically dlopen a big lot of plugins (see my manydl.c program, with enough patience it can generate then load nearly a million of plugins). So your XML thing could drive the loading of plugins. Your multi-component / multi-connector description reminds me of Qt signals and slots (which requires a moc preprocessor; you might need something like that too).

Most C++ implementations use name mangling. Because of it, you'll better declare as extern "C" the functions related to plugins (and defined in them, and accessed by dlsym from the main program). Read the C++ dlopen mini HowTo (for Linux at least).

BTW, Qt and POCO are C++ frameworks providing some portable and higher-level approach to plugins. And libffi enables you to call functions whose signature is only known at runtime.

Another possibility is to embed some interpreter, like Lua or Guile, in your program (or write your own one, like Emacs did). This is a strong architectural design decision. You may want to read Lisp In Small Pieces and Programming Language Pragmatics for more.

There are variants or mixes of those approaches. You could use some JIT compilation library (like libgccjit or asmjit). You could generate at runtime some C and C++ code in a temporary file, compile it as a temporary plugin, and dynamically load that plugin (I used such an approach in GCC MELT).

In all these approaches, memory management is a significant concern (it is a "whole program" property, and what actually is the "envelope" of your program is "changing"). You'll need at least some culture about garbage collection. Read the GC handbook for the terminology. In many cases (arbitrary circular references where weak pointers are not predictable), the reference counting scheme dear to C++ smart pointers might not be enough. See also this.

Read also about dynamic software updating.

Notice that some programming languages, notably Common Lisp (and Smalltalk), are more friendly to the idea of runtime importing functions. SBCL is a free software implementation of Common Lisp, and compiles to machine code at every REPL interaction (and is even able to garbage collect machine code, and can save an entire core image file which can be later easily restarted).

Answer 2

Clearly you're trying to roll your own style of Simulink or LabVIEW type software, but with an unholy XML component.

At its most basic, you are looking for a graph-oriented data structure. Your physical models are built up of nodes (you call them components), and edges (connectors in your naming).

There's no language enforced mechanism to do this, not even with reflection, so instead you'll have to create an API and any component that wants to play will have to implement several functions and abide by rules set forth by your API.

Each component will need to implement a set of functions to do stuff like:

• Get the name of the component or other details about it
• Get the number of how many inputs or outputs the component exposes
• Interrogate a component about a particular input our output
• Connect inputs and outputs together
• and others

And that's just for setting up your graph. You'll need additional functions defined to organize how your model is actually executed. Each function will have a specific name and all components must have those functions. Anything specific to a component must be reachable through that API, in an identical manner from component to component.

Your program should not be trying to call these 'user defined functions'. Instead, it should be calling a general purpose 'compute' function or some such on each component, and the component itself takes care of calling that function, and transforming its input to its output. The input and output connections are the abstractions to that function, that's the only thing the program should see.

In short, little of this is actually specific to C++, but you will have to implement a sort of run-time type information, tailored to your particular problem domain. With each function defined by the API, you'll know what function names to call at runtime, and you'll know the datatypes of each of those calls, and you just use regular old dynamic library loading to get it done. This will come with a fair amount of boilerplate, but that's just part of life.

The one C++ specific aspect you'll want to keep in mind though is its best to have your API be a C API so you can use different compilers for different modules, if users are providing their own modules.

DirectShow is an API that does everything I described and can be a good example to look at.

Answer 3

You can use leadelf library.so then cross check the symbols against the library header. Here is a shell php script doing that https://github.com/comarius/elf_resolver