If I can try to read between some lines of what you said and try to deduce where you're trying to move with the design, I believe I ventured down similar rabbit holes and came out cursing with more gray hairs.
In my particular case I wanted to abstract the rendering capabilities of all these rendering backends (OpenGL, OpenGL ES, DirectX, consoles, etc) in one abstract rendering interface to rule them all, like IRenderer, with the hope of minimizing porting grief by simply swapping one renderer subtype with another (ex: RendererGl with RendererDx). That's a bit different and probably more ambitious than what you are doing but I suspect there is a common theme to our design headaches.
What was fundamentally so difficult to design with that route beyond the different rendering capabilities between all these backends and the difficulty of trying to design a uniform abstraction for all of them was how I was trying to make the renderer oblivious about the rest of the game engine and world. I was trying to decouple the renderer away from everything else and have the game engine have knowledge of how to use the rendering interface instead of having the renderer have knowledge about the engine.
To put it simply my renderer interface was huge like:
class IRenderer
{
public:
// boatload of functions related to rendering along
// with auxiliary abstract interfaces like ones for
// vertex buffers and textures.
};
And it was completely oblivious of higher-level abstract concepts in the engine like particle systems, models, bones, materials, lights, etc. Now I did still decouple the game logic from rendering code by injecting and associating renderable/rendering components to them, but fundamentally I was trying to make the renderer oblivious of the game world/scene.
What I found to be so, so, so much easier was to make the renderer have access to and understand the entire game world/scene and all the abstractions within. That made my renderer interface so simple like this if we use a more object-oriented approach (these days I use an ECS):
class IRenderer
{
public:
virtual ~IRender() {}
virtual void render(const IGameScene& scene) = 0;
};
And that does mean I have this epic implementation of that function to render scenes (though decomposed into various smaller functions and medium-sized objects), and it does mean my renderers cannot possibly be made to work anymore for any engine other than my engine. Each concrete renderer has to know how about the abstraction of my specific engine's particle system, for example, and how to render it is up to the concrete renderer to use whatever low-level mechanisms it needs to do so.
If you do it that way, then you can be as abstract as you want with the rest of the game engine and just think in terms of high-level logic without wrestling with the most effective to render the data and store and create and manage buffers and bind shaders and textures and so forth. That's all handled within the renderer itself.
For things like models which are loaded from a file, the renderer might encounter an abstract model in the scene which isn't loaded, load the file, then externally associate a VBO to that model to be used for subsequent rendering. All the gory details of rendering are inside the concrete renderer.
And that's still not easy to implement the renderer (it does help if you can generally attach data, like a component, to any entity that already exists, like a VBO component), but it was way simpler than what I was doing before as far as getting designs correct (the implementation was still hard) and I stopped having to fight with my own designs and interfaces trying to headbutt them into shape. These kinds of concerns you listed become rather moot in that case (in the sense of not tempting you to change your central designs) because you'll have all the breathing room you need in a particular renderer's implementation to get knee-deep in the specifics and peculiarities of OpenGL or any other backend to render the abstract objects/entities that make up your game world in an efficient way.
There's very generalized design sort of lesson here. If something, A (a subsystem, interface, collections of interfaces, say high-level game engine abstractions) is relatively easier to design correctly, while B (say the renderer) is extremely hard to design correctly, then consider not making A depend on B. Instead of like:
See if you can invert the flow of knowledge and dependencies and have B depend on A, like so:
Now the dependencies aren't flowing to B and you have all the breathing you need to change it without changing any central designs (a "central design" would be something lots of things depend upon which is extremely costly and difficult to change). Basically don't make things that are so hard to get right difficult to change. As straightforward as that seems, I think it's somewhat humanly counter-intuitive because I don't think most of us instinctively tend to abstract and flow dependencies in the direction of what's easiest to design correctly. It tends to be more intuitive to create abstractions in a way that yield the smallest implementations behind each abstraction and direct the dependencies that way instead of directing the dependencies away from what's most difficult to design correctly. Yet smaller implementations be damned if the designs on top constantly tempt us to reconsider them.
