Micro-Facets of Functionality
This is going to be more of a general design strategy answer, without understanding the precise issues you are dealing with in a networking context. It might completely miss. Yet understanding this was so helpful to me in just software engineering in general that I'm eager to share it, and I think it has a good chance of helping. Simplifying state/resource management and control flows tends to go hand-in-hand with simplifying multithreading, which is handy.
To me the perceived sense of overwhelming complexity of a very large-scale codebase is often owed to the fact that it's structured in a way that has us zooming in to a micro-level of teeny islands of functionality and control flows and then trying to have us reason about how it functions abroad at an overseer level. In that case, it's making us try to sum up little pieces of functionality to try to reason about the codebase, and it's hard to try to work bottom-up in this fashion and piece together what's actually going on.
This problem is multiplied when you have a central data structure, with all these little teeny micro-facets of functionality all weaving and winding and forming a maze of functionality which ultimately works their way towards calling functions which update this central data structure.
Such a codebase can be perfectly SOLID in its engineering. All entities might perform a singular responsibility, all dependencies flow towards abstractions, everything is easily extensible that needs to be extensible, etc.
Regardless, the conceptual complexity of the codebase seems beastly when trying to comprehend it, and trying to achieve both thread safety and thread efficiency might seem like a hopeless endeavor. It's because this twisting maze of control flow still makes it very difficult to reason about when and where critical things happen. Such systems make us have to zoom in, making us trying to understand the big picture from little puzzle pieces.
"When does this shared, central data structure get updated?"
"Well, it might be updated when this event is triggered when the
user pushes this button which causes this function to get called which
causes that function to get called which updates this structure. It
might also get updated when this event is triggered which calls
functions A->B->C->D->E....->Z which then updates this structure, etc. etc. etc."
It's hard to reason about when and how this shared state is accessed and modified, and when it's hard to reason about that, the temptation is often to lock with every little hand that touches it.
Flat, Bulk, Deferred Processing (Mass Consumer Design)
The solution to me here to both alleviate perceived complexity and also kind of flatten the whole system and make it easier to multithread efficiently with smarter locks is to design the system in a way that allows you to zoom out. Instead of this:
We do something like this:
By doing this, you centralize one single place in the entire system that can actually modify this data structure. And that one place can potentially multithread to its hearts' content, using one of the simplest means of multithreading -- the parallel for loop. Such a place then can process many requests at once, and thus lock at a more appropriately coarse/granular level. It'll also make it easier to split up the central data structure, for example, and reduce the contention that way with only the central processor to update in response.
Just about everything is easier when you avoid having a teeny little isolated pipeline for each little request that can occur, and instead have a bulk, centralized pipeline that handles multiple requests at once. Just at a meta level, reasoning about such a codebase's behavior starts to become like reasoning about a dozen continents rather than thousands of little islands.
In such a case, each micro-facet (each client, e.g.) might push or write some local state which is then read/popped/cleared by this deferred, flat, bulk processor. A general approach is have each one push to a local, concurrent queue (producer) that gets popped by the bulk processor (mass consumer), though a queue is overkill if you can just have some atomic variables you can read/write.
We have now identified a number of problems in fringe cases like calls
during initialization or shutdown of the server and/or tunneling app
and I'm struggling to properly fix these as they are mostly related to
multithreaded object and thread lifetimes and just throwing more
mutexes or more locks at the problem really doesn't cut it.
By having one centralized place which does all the heavy-lifting, it's easier to multithread but also avoid issues like these associated with attempts to update shared state once destroyed or prior to initialization. This kind of awkwardness with initialization and destruction order can also be understood and tackled from untangling state management and associated control flow. The central, mass consumer (bulk processor) would simply ignore the local state of clients (not consume and therefore not do anything with shared state) when the system is not ready (pre-startup or post-shutdown). The clients are still free to safely push/write local state, since they are not modifying any shared/central state in doing so. The centralized system will pull/process that state when it's actually ready to do so.
If this kind of asynchronous, deferred processing is applicable, it can tremendously simplify all kinds of goals ranging from correctness to simplicity to safety and efficiency. Most of all, codebases structured like this just tend to be easier to comprehend for humans at a broad level without thinking so much at a micro-facet level, the "what if?" scenarios become few in number, and life becomes a lot simpler.
If there are multiple phases involved (like a state machine), then you can have more than one centralized bulk processor. The first could push results back to each entity ("micro-facet"). The second system then consumes those results and processes the second phase in a serial fashion for order-specific, multi-phase processing.
Optimization
To avoid needlessly looping through micro-facets which don't need to do anything, one strategy is to aggregate them into blocky data structures, like so:
This aggregates 4 "micro-facet" entities into a each unrolled block (kind of like an unrolled linked list but without shuffling contents around to keep nodes at least half full). You can keep track of vacant spaces in the list and reclaim them upon subsequent insertions. And of course you can aggregate more than 4 of these, maybe even like 64 at once, and can even form a hierarchy (I just used 4 because it's easy to draw).
The key idea is that you can skip N (4 in this diagram) micro-facets at once to process if this Needs Processing atomic flag is not set, so it's kind of like a miniature skip list (and can be expanded to a full-blown hierarchy for maximum scalability). Any time one of these entities have work that needs to be done, they can set the Needs Processing flag in the block they reside in (and possibly upwards to the root of the hierarchy if a full-blown hierarchy is used). We can also use condition variables to put this deferred processing thread to sleep when there's nothing to do.
The bulk processor works its way downwards from these blocks that have Needs Processing set to true down to the individual entities that need to be processed within that block.
CES/ECS
This is similar to the design of entity-component systems used in games, but playing especially to the "systems" part. Systems as bulk processors can really help reason about the software, as well as multithread more efficiently when each system is doing a lot of work. Even games spanning millions of lines of code might only have a couple dozen systems which are the sole places doing the critical work, and that's really easy to digest for the human brain when there's only a couple of dozens of places in the codebase that are responsible for the most critical updates to the software's core state, instead of tens of thousands of miniature functions to think about.
The key is not to confine yourself into a little tiny island of functionality, and to avoid all such little teeny islands communicating directly with central data structures. Make it one big island of bulk processing that has sole access to the central data, make each little island which is responsible only for its local state, and life will become a lot easier.
The control flow will become untangled in the places where it's most critical, and likewise the temptation to lock on every little teeny request will also go away. Your call stacks will also become very shallow. Instead we have end up having big, chunky bulk work to do in a handful of centralized places, and that leaves a lot of breathing room to lock at more appropriate levels, and less frequently with less contention. Most importantly, it should have you dealing with bugs a lot less when the question of "when and where" the critical things happen, like updating central data, is no longer such a mystery that requires reaching for a magnifying glass and sifting through the entire codebase.
