I have a rather simple (i thought) setup that causes me some grief.
I am using a queueing system (AWS SQS) and a few worker docker containers with code written in .net core that pull from a queue and process messages. Multiple workers can pull from a queue at the same time.
Certain operations belong to a single job. E.g. we get 100 units to process, so we can create a
job in our SQL db and then send 100 messages to SQS to process. When all workers are done, and only once, we want to delete the job and message some external service that we're done.
It is crucial that we message the external service only once, because the service would result in some user-visible activity.
What is happening is that often the last couple of units finish processing almost to a millisecond at the same time, so we are getting either duplicate messages going out (if we message first and then delete the
job record) or various exceptions if we try to delete the job concurrently.
What is the most appropriate mechanism to handle the locking here? One thing we tried is adding a random wait times when we know we are close to finishing the job, but those don't seem to work reliably.