Design choice to avoid lock contention

Question

I have a network application F. It receives requests from one (or many) client network function. F can handle multi-client requests using an epoll loop. F maintains a state machine for each client(user). F also maintains some context for each user or client. Whenever a message is received from a client network function (by one user), F processes the message and if required fetch context specific to this user and updates the context.

Currently, the contexts of different users are maintained in a C++ STL map.

map<clientId,context> userMap;

where clientId is an integer and context is a struct containing user-specific data. Whenever I need to access the userMap I would take a lock first, access the data, and then unlock.

For example, if a client sends a request message X. The server F, epoll_wait() will get the event (i.e. the incoming message). Then this messages is read from the socket and processed further. In the server handleX() method will be invoked (if user state and context is consistent and allows for handleX to be executed for this user). Each of these handleX function needs the current user context for some computation. it locks the userMap, gets or set the data, and then unlocks it.

In the single threaded version of F, a single thread only waits for events and processes those events one by one.

I tried to use a thread pool to check the multicore scalability of F. In this case, a single thread reads the messages from the socket and puts those messages in a queue. A pool of threads are waiting on the queue and picks any messages that pushed on the queue. But the throughput is not better than single-threaded code version of F.

I think locking is inherently serializing the F's code. I would like to know is there any other model of user context store and retrieve to minimize lock contention?

Answer 1

Since you are doing a performance benchmark (comparison), it helps to verify whether your conclusion is supported by the experiment.

1. If single-core CPU is the bottleneck, you should see high single-core CPU usage. (Use thread-core pinning to keep the thread on one CPU core; use a CPU usage tool that shows per-core usage statistics.) If this is not the case, the load generator does not generate enough messages to keep even a single CPU busy. Consider: increasing the load; use more than one machine (on the network) to generate the load if necessary.

2. A simple way to measure existence of lock contention (i.e. qualitatively, not quantitatively) on lock-protected data structure or mutex is to add a try_lock in front of every lock (explicit or implicit), and record the result of the try_lock. Count the number of successes and failures using two std::atomic integer counters. This gives a qualitative evaluation of whether lock contention is likely or unlikely to be a cause of performance issue.

3. Make sure the "queue" you are using are intended for multi-core job distribution purpose. You should use a concurrent queue (not merely a multithread-safe queue) if possible.

4. Similarly, you should use a concurrent hash map (again, not merely a multithread-safe map) if possible. Consider using std::shared_ptr for the map's value (the struct containing the user context), so that the user context's lifetime does not go away while the code is executing. (That may require some other code to switch to std::weak_ptr to tackle circular references.)

5. To achieve (3) and (4), you should be using a concurrent data structure library, such as Intel Thread Building Blocks (TBB), or Microsoft Parallel Patterns Library (PPL) if the code only needs to run on Windows. There may be newer, open-source, less-encumbered-licensed ones. The newer libraries might even be more performant, as they are optimized for the newer generations of multi-core processors. (TBB and PPL are at least a decade-old.) However, be sure to read the documentations carefully, and judge the level of expertise of the authors. A widely-used implementation (especially if used by other well-known, well-maintained software projects) may be more trustworthy. The risks of using lesser-known implementations are bugs, especially severe ones, and claims of performance or lock-free that are not substantiated in reality.

6. After making these changes, run the single-threaded / multi-threaded performance comparison to see if anything changed.

7. It is possible that the code change may degrade performance. In that case, you will want to be able to revert to an earlier version of the code. Thus, it is good idea to use version control system, such as Git, Mercurial, SVN, to name a few.

This is just some off-hand remarks. There are lots of other possibilities, and lots of things to take care of.

Answer 2

You could put a reference to the client's context on the queue together with the message, so that the map would be completely managed by the single thread. This is assuming that all clients will have a single context, and it won't be deleted or replaced by a new context, but just its contents will be updated.

With this there is still a potential race condition if you get multiple concurrent requests with the same clientId, so a mutex for each clientId would be needed -- perhaps make the context itself thread-safe?