Ensuring Thread Execution Order when Writes Must Occur Before Reads

Question

I'm currently designing a server with the following structure:

• A TCP thread pool that receives data from the network
• A queue that holds these requests
• A worker thread pool of a fixed size that takes requests from the queue and executes some work where a ConcurrentHashMap is read, and results analyzed. After the work is complete, the results are posted to a write queue.
• The write queue stores write requests for ConcurrentHashMap
• A write thread pool of a fixed size that takes requests from the write pool and writes them to the ConcurrentHashMap

Server constraint:

• For a particular entry in ConcurrentHashMap, a write will get posted to the network several seconds before a read gets posted to the network. The threads must never allow the read to get scheduled before the write occurs.

My ideas so far:

• First, I'm hoping that the 3-5 seconds or so is a big enough head start for a write to get through before the read
• Set the write threads to have high priority and the read threads to have low priority
• A read can check the last write time, and if large enough (like several minutes or hours ago), it knows to ignore the last write because it is out of date. However, it then knows that it can't make an informed decision. I could have the thread retry like twice in the case of an outdated entry, but this feels clumsy.
• Added the write queue so that the TCP pool can post directy to the write queue (write requests don't need to be processed by the worker pool) instead of the job queue while read requests are put in the worker thread queue, and then post some logging info to the write queue later. after the worker pool

Are there any ways to ensure that reads have a low chance of occurring before a write or are the precautions I've already taken sufficient? Should I use the read retry mechanism?

Also, is a write queue actually needed or can I just use worker threads more generally to also handle writes? Write order is not important.

Answer 1

Don't mess with time. They're fickle to work with and error prone in a distributed system, especially because you'll need to synchronise clocks and deal with latency. Instead use a version number. For every write request, increment the version number on the data; with every read request, send a version number to conditionally perform the request, if the read request's version number does not match the current version of the resource (or, if your application can tolerate reading from future versions, then loosen the matching criteria so it's a less-than match), then reject the read request (the requesting client can then wait for a few second and resend the read request) or put it back to the read queue and wait until a suitable write request arrives.

For extra reliability, your read queue can be a priority queue. The version number is the priority, if the front of the priority queue (the item with the lowest version number) is higher than the current version of the resource, then the read thread should block until either the write thread wakes it up due to a write event, or its woken up by an enqueue to the read queue that have a lower version number than any other entries in the queue (i.e. it's been inserted to the front of the priority queue).

Answer 2

I do not fully understand what your application does but your concerns and approaches seem off.

• Do not mess with thread priorities, these will give you nothing.
• If you do not want reads and writes to get in each others way, just lock on the queue object.
• a read thread looking in a queue to find nothing is there (yet) is not a bad thing. Trying to control the order in which different threads run is pointless and defeats the purpose of threads.
• do not do work within the lock, only lock while using the shared resource, that is write to the queue or read from the queue.
• Put the read thread to sleep for a while before looking for new items again.

Answer 3

First, I'm hoping that the 3-5 seconds or so is a big enough head start for a write to get through before the read

It's really a bad idea to assume something like this. You should just write your code to deal with the fact that a read may show up first.

So if data is stale, you want the read operation to wait until it's updated, is that correct? You might want to use Condition objects. If the data is stale, create/reset a Condition object that the write operation will signal when the data is updated.

Putting threads to sleep can work but has some tradeoffs. For example, how long does it sleep? If your sleep is too short, the thread spins doing nothing but taking up CPU time. If your sleep is too long, your application is less responsive.

Using a lock, semaphore, or Condition object is simpler to code and more responsive (the read operation just blocks until there's something useful to do.)

Also, check out the book Java Concurrency in Practice. Concurrency is really hard to get right and this book discusses a lot of unexpected gotchas and how to do all of this stuff correctly, including how to test it!