Over and over I am faced with a similar problem: I have to perform two actions that are mostly unrelated, except that they need to share a mutex lock, at least for a moment. For example:

void action() {

//Somewhere far away:
std::mutex p;
void doStuff() {
    std::unique_lock lock(p);

//In some other, distant corner of code:
std::mutex q;
void cleanup() {
     std::unique_lock lock(q);

Now, when this is all compiled, it produces a sequence of operations like this:

void action() {
    lock p
    ... //doStuff
    unlock p
    // <-- bad things can happen here
    lock q
    ... //cleanup
    unlock q

The problem is the gap between unlocking p and locking q. It is unlikely, but something bad could happen right between those two. I would like to have those happen in the reverse order: first lock q and only then unlock p. Is there a way to organize code, a pattern perhaps, so that it could be achieved, while maintaining the code of doStuff and cleanup decoupled as much as possible?

Edit: you can assume that doStuff and cleanup are not going to be called in reverse order. In actual code this is not possible to do.

To give a concrete example of this happening: I have a class managing a file-based cache system, and it needs to "catch" a mutex/atomic from within the implementation of std::shared_ptr:

(possible implementation in std)
template<class T>
class shared_ptr_control_block {
    atomic_int refCount;
    atomic_int weakRefCount;
    T* value;
    std::function<void(T*)> deleter;
    void decrement() { //invoked by shared_ptr destructor
        if (--refCount == 0) {  //this is my (implicit) p lock

class Cache {
    std::shared_ptr<Data> get(std::string filename) {
         std::unique_lock lock(m);
         std::shared_ptr<Data> result = data_[filename].lock();
         if(!result) {
             Data* ptr = new Data;
             openActualFileAndReadIntoDataPtr(filename, ptr);
             result = std::shared_ptr<Data>(ptr, [this,filename](Data* ptr){
                 std::unique_lock lock(m);  //this is my q lock
                 storeDataPtrBackToFile(filename, ptr);
                 delete ptr;
             data_[filename] = result;
         return result;
    std::mutex m;
    std::unordered_map<std::string, std::weak_ptr<Data>> data_;

Cache is supposed to be thread-safe: Multiple threads can request the Data under the same name. One thread will open the actual file, and the second thread will reuse the same data pointer. Either thread may modify Data (it is Data's job to maintain its thread-safety) and when the last thread destroys its Data shared pointer, it is again saved to a file.

However, there is a subtle gap: when refCount is decremented to 0, but before shared_ptr's deleter is called, another thread may request the same file. What may end up happening, is that the corresponding weak_ptr will recognize the pointer as expired, and Cache will reread the old version of the file - before the deleter managed to save the newer file version.

A straightforward solution in this particular case is to convert the refCount atomic to {mutex, int} pair, and have the deleter invoked while the mutex is still held. As soon as Cache::m is locked, the refCount's mutex could be released, but to achieve that I would have to actually pass the lock from shared_ptr into the deleter, so that it could be unlocked.

This straightforward solution practically requires reimplementing the whole shared_ptr. Not too difficult, but time-consuming and head-scratching with a question lingering - "isn't there a better way?"

2 Answers 2


You're asking multiple questions.

For the general case of calling multiple decoupled functions with correct mutual exclusion, the solution is to

  1. Expose the mutex associated with each function
  2. Split the function into
    • an outer, which gets std::unique_lock as now, and calls
    • the inner, which performs the work
  3. Have your compound operation use std::scoped_lock to acquire all the locks
  4. Call the inner functions, while all locks are held

If you're doing this a lot, you can do a neat job of bundling these functions into Lockable objects with CRTP and automating the pattern.

Your second question, about the cache specifically, is more easily solved with a small change: have the deleter remove the weak_ptr from the map. Now you have three easily distinguished cases:

  1. Key isn't in the map at all, so open the file and add it

  2. Key is in the map, and locking the weak pointer succeeds: no change

  3. Key is in the map but locking the weak pointer fails: the file is about to be written out in another thread and you can't do anything to stop it (you can't prevent the control block being deleted, for one thing).

    That means you need to temporarily release the mutex (so the deleter can run), wait for it to finish, and then reacquire the mutex and reopen the file as for case 1.

    This is exactly the pattern of waiting on a condition variable. So, add one of those and have the deleter notify it after removing the map entry.

Actually the easier solution is probably to change your cache into an LRU cache and write files out not when the last concurrent user finished, but when the next new file is requested. Since both happen under the same lock this is sufficient to fix the race condition.

However this changes the behaviour of your cache while the solution above only fixes the specific problem.

  • The latter question was more as a concrete example of the former, thank you for a different approach to it though! However, with the former, general question - if I understand correctly, you suggest to lock both locks for the full duration of both functions? My goal is to lock q and then unlock p right after that. I don't want to lock both p and q during full execution of both functions.
    – CygnusX1
    Nov 4, 2023 at 10:56
  • Locking both for the full duration is the only way to avoid the problem you identified, though. If that leaves the locks held for too long, you need to refactor the work so that more of it can be done without holding the lock in the first place.
    – Useless
    Nov 4, 2023 at 13:20
  • Specifically your suggestion doesn't work in general because you can't prevent deadlock between two threads calling the functions in opposite order to one another. Generalized deadlock-proofing requires the locks to be acquired always in the same order, not in the order you arrange your function calls.
    – Useless
    Nov 4, 2023 at 13:24
  • Why do we need to expose the locks? Can't we introduce a third mutex to handle higher level operations without invasive changes to low level components?
    – Basilevs
    Nov 4, 2023 at 23:06
  • If you can change p & q to both use the same mutex, that also sidesteps the problem. It's still a change to the low level components though. You can't just add a third mutex unless you know the low level components will never be used without it
    – Useless
    Nov 5, 2023 at 0:00

General question from the first part is solved by using a third mutex, that is always captured before working with any of lower level components. This ensures proper locking order and safety without invasive changes in lower level components.

Surprisingly, the example problem from second part has nothing to do with concurrency!

The abstraction leak you experience is caused by exposing the implementation detail (shared_ptr) to clients of the component.

Instead of a shared_ptr, return a wrapper object that knows about Cache's mutex and handles weak dereferencing (capturing the mutex before disposal or dereference). Note, that in your context you are unlikely to need the full power of shared_ptr in client code, so expose the minimal interface client needs and keep implementation simple.

  • After shared_ptr is obtained from the cache, it is passed into multiple algorithms that need the functionality of shared_ptr and do not do anything with cache. Passing a CacheDataWrapper instead of shared_ptr - that wold be a Cache implementation leak into those algorithms.
    – CygnusX1
    Nov 5, 2023 at 8:36
  • The details of the cache are completely separate from the details of the compound locking problem, and follow up comments suggest that neither are adequately specified in the original question. Can you move one of the two into a separate post, and include your actual requirements?
    – Useless
    Nov 5, 2023 at 13:17
  • @CygnusX1 could you clarify what kind of algorithms require type of shared_ptr and are not compatible with dereferencable concept in general?
    – Basilevs
    Nov 6, 2023 at 9:27
  • @Basilevs In essence, you are asking why shared_ptr exist? (because if there are no algorithms that require shared_ptr, there is no reason for shared_ptr existence). Of course I could replace shared_ptr itself with my own custom reimplementation of it, with some tweaks. In fact, that's what I did - as stated in the question!
    – CygnusX1
    Nov 6, 2023 at 12:24
  • @Useless Frankly, I didn't want to ask specifically about the Cache, so I didn't put a separate question for that. The main question is a result of me trying to abstract a problematic pattern I am recognizing. This abstraction of mine is not without its flaws, I admit, hence some aspects remain vague.
    – CygnusX1
    Nov 6, 2023 at 12:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.