5

How does one best write tests for a multi-threaded producer-consumer C program?

I know GDB can be used to debug threads, but I believe GDB also stops the thread currently being debugged while other threads continue running.

I want to test proper mutex lucking/unlocking and blocking by my threads. I've looked around a lot but haven't been able to find any good resources.

I have unit tests for all the non-concurrent functionality, but want to make sure my concurrency is correct. How can tests for multithreaded programs best be achieved?

1
  • Write unit tests with multiple threads. Be sure to track failures on all threads, let one main thread report those results back as final success/failure for the test case. Also, write test apps using multiple threads and use tracing (ie: printf style debugging) to get constant output from all threads (should be prefixed by thread id time debugbit function name etc). Stepwise debuggers aren't great for diagnosing threads
    – ebyrob
    Oct 12, 2015 at 20:59

1 Answer 1

6

The only way that I know of for testing software that utilizes concurrency of the locking kind is by stress-testing them.

You can write unit tests that cover some or even all of the operations that involve concurrency, but you have still not really tested your software, because the problems with concurrency come from unexpected interactions between threads which are highly dependent on timing, and therefore non-deterministic.

So, you need to write one test which runs your system for hours on end, hoping that if there is a problem, it will manifest itself.

Of course, your software may contain a defect which manifests itself only once in a million lock operations, in which case, from a statistical point of view, you stand virtually no chance of detecting it during in-house testing, so you ship your product, and thousands upon thousands of users start using it day in, day out, around the clock, and the defect starts manifesting every so often, which is a nightmare.

That's why the industry has been moving away from concurrency of the locking kind, and uses other approaches instead, like share-nothing systems where the threads only communicate with each other via message queues, through which they pass only immutable data, so that the only place in the entire system which involves locking is a few instructions in just one class, the message queue class. The great benefit of these systems is that they are testable, whereas concurrent systems of the locking kind are not really testable.

They are more like "stress-test and pray."

3
  • 3
    While the specifics escape me at the moment, I recall threaded UI code (swing workers and all that) code that when built for dev or test environments had random sleeps tossed in the threads. This uncovered an astonishing number of race conditions and deadlocks. I absolutely agree that locking based concurrency is a nightmare and messages and immutables being the exchange to be the right approach.
    – user40980
    Oct 12, 2015 at 21:49
  • 1
    This is the approach I use, yes. Since I usually use pools of inter-thread comms objects, I can easily spot if any get lost or get released back to the pools twice. Typically, a test harness, or auto-test, or multiple clients or whatever shove in data as fast as it will go and I leave the thing runnng over the weekend. If I take the load off on Monday morning, the app is still working and the pool has neither lost or gained level, I am happy that the app has sufficient quality, (multithread-wise), to ship it:) Dec 10, 2015 at 19:52
  • And yes, the crazy 'locks everywhere' apps are pretty well hopeless, as are those with continual thread create/terminate/join, and all those written by developers who have been influenced by the 'you MUST always terminate all your threads gracefully and clean up everything' mantra. Dec 10, 2015 at 19:56

Not the answer you're looking for? Browse other questions tagged or ask your own question.