2

I have a rather simple (i thought) setup that causes me some grief.

I am using a queueing system (AWS SQS) and a few worker docker containers with code written in .net core that pull from a queue and process messages. Multiple workers can pull from a queue at the same time.

Certain operations belong to a single job. E.g. we get 100 units to process, so we can create a job in our SQL db and then send 100 messages to SQS to process. When all workers are done, and only once, we want to delete the job and message some external service that we're done. It is crucial that we message the external service only once, because the service would result in some user-visible activity.

What is happening is that often the last couple of units finish processing almost to a millisecond at the same time, so we are getting either duplicate messages going out (if we message first and then delete the job record) or various exceptions if we try to delete the job concurrently.

What is the most appropriate mechanism to handle the locking here? One thing we tried is adding a random wait times when we know we are close to finishing the job, but those don't seem to work reliably.

1

The typical way to handle something like this is to have each worker post a "done" message to another queue, then have an additional task that only waits for all the done messages before cleaning up the job.

Another way that's a little more coupled to your database implementation is to use the database guarantees to do your synchronization. For example, in cassandra, you can do a DELETE IF EXISTS, which will notify the caller if it tried to delete but it was already deleted. Then, only the successful worker would do the notification step.

1

At it's heart your question is about one of the tripping blocks in designing asynchronous distributed systems: knowing when a batch job is complete. There are lots of ways you can attempt to work around this that aren't terribly reliable. Here are two solutions that are robust:

The first answer is that you redesign the system so that it is not asynchronous. Get rid of the queues and just do the work. You can run more than one threads if that is deemed necessary. The downsides are that this could mean a significant amount of rework and redesign and it can be harder to scale.

The other answer is that you need a watcher process. It knows how many tasks need to be executed and verifies when they are all complete. It's this process that would manage any pre or post-batch work. This allows you to leave the distributed queues in place but you will need some sort of mechanism for the watcher process to be notified about the completion of each task. One slick way of doing this is a key part of the Apache Storm project. The basic idea is that you give each task a unique id. When the watcher submits an ID, it XORs that against a value which starts at 0. When the task is complete, you XOR it's ID back into that value. When all the tasks are done it that value will be restored to 0. Of course you could always just keep a ledger and that will tell you what is still outstanding.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.