In my opinion you should just design a proper API for your clients to call while they compile plugins in the form of a dylib which gets loaded and invoked by your application.
It does require spending more time in advance to design your API/SDK but you can use it yourself to build your application, basically bootstrapping and building your own software with the same API and SDK that your customers use to build plugins for it. You give your clients headers indicating the interfaces they can use (ex: tables of function pointers) and pass them a pointer to access these interfaces when their plugin is loaded and an entry point function exported by their plugin is invoked. Only header files for public interfaces are exposed to your clients. The rest remains hidden. Simplistic example:
#ifndef SDK_H
#define SDK_H
// Central interface for your software development kit.
struct Sdk
{
// A function available for your customers to call in their plugins.
void (*do_something)(int x);
};
#endif
// Inside the plugin code, compiled separately away from your
// application with no access to its source code, only the
// header files for the SDK.
#include <sdk.h>
// This exported function inside the plugin gets loaded and called
// by your application when it loads this plugin, passing it a pointer
// to the SDK interface required to invoke all the functionality
// publicly available for plugins to call.
EXPORT void my_plugin_function(struct Sdk* sdk)
{
// Make your application do something from within this plugin.
sdk->do_something(123);
}
On top of this I'd actually use a C API (though you can implement the C functions in C++). C is the most widely standardized language for APIs. If you use C, you can even call those C functions from a wide range of programming languages and allow your customers to, say, implement their plugins in C#. They might then create like a YourSoftware.Net library which allows your C API to be used in a way that is idiomatic to the .NET languages and framework just as people have done with, say, OpenGL. This isn't the case so much if you create a C++ API which is subject to more vendor issues which lack an ABI standard, name-mangling of exported symbols for function overloading which could be very compiler-specific and fugly to import at the very least, exceptions which can't be safely thrown across module boundaries, type capabilities like copy ctors and dtors which some languages won't know what to do with, etc.
Likewise you can build a static C++ library on top which wraps the C API and turns it into idiomatic C++ code (ex: translating error codes into exceptions, turning C resources which have to be freed manually into RAII-conforming C++ objects with destructors, function pointers into virtual functions, etc).
This does take time and thought, especially if it's your first time doing it, but it is the primary way that extensible plugin architectures are created, and the benefits of doing this properly, if it is actually needed, often outweigh the costs in the long run.
Another option is to embed a scripting language into your application, like Lua. Then your clients can write Lua code which calls functions in your application at runtime, and you can even have a scripting console directly inside of your software where your customers can type in code and run it on the fly without restarting the application. That also carries its share of advance work though that tends to be as costly or even more than designing a C SDK, but it does let you do some really cool things with your software at runtime and can actually aid a lot in testing its functions at runtime on the fly even if you prefer to write the majority of the code in C++.
I'm concerned about mixing debug and release code, though, as if I
deliver debug files perhaps my source code and implementation will be
available to them but it I deliver only release mode .lib files then
perhaps they won't be able to debug without issues.
I think this is an impractical concern. If I'm writing plugins for Google Chrome, I'm not interested in debugging Google Chrome's codebase, only the code for my plugin. Most likely your customers won't be so interested in wading through the implementation details of your codebase just as you aren't interested in letting them see it. The usual way is to give the clients release binaries. They write plugins and can build debug versions of those and debug and trace through their own code extending your application's capabilities, but don't bother with debugging your code and tracing into the API calls.
