I'm developing distributed system and trying to use best practices of microservice architecture. I was faced with a situation when I think I need something like distributed locks. Since I have not so many experience in microservices and still not sure about the final solution. I need any suggestions, thoughts, best practices, possible articles that can help me (now I'm investigating the issue and already have possible solutions).

I'm using Azure Service Bus to organize pub \ sub communication and Redis as shared cache in my system.

Here is an abstract schema of what I actually trying to do . Explanation: I have queue with incoming data (dataFlow). The format is the following {dataOwnerId: "1", dataOwnerSubId: "1", data: "status"} The dataOwnerId is an aggregator for a 1-n sub owners. So it's 1 to many relationship.

The owners periodically (let say each 1 minute) push new status (data) to the queue related to certain sub-owner. So if the dataOwnerId related to dataOwnerSubIds [1, 2, 3]. The message can be {1, 1, status} or {1, 2, status} or {1, 3, status} and after 1 minutes {1, 1, status2} or {1, 2, status2} or {1, 3, status3} etc

The queue has 1-n consumers (dataConsumers) which processed data from queue and storing to DB. Data order is non-deterministic, the first available consumer takes the first message from queue. So the dataConsumer1-N can handle the data from owner1-N and sub-owner1-N.

For example in one time the dataConsumer1 can handle the data from owner1 and sub-ownerN the next time he can handle the data from ownerN and sub-owner1.

Requirements: The possibility of auto adding new data owners and data sub-owners to the system during queue processing required.

Possible solution: I can have situation when the dataOwnerN + 1 (new customer) already push data to the queue but he is still not in the system. In this case the dataConsumers will skip this new owner (skip storing the data) and push message NewOwnerEvent to ServiceBus.

The messages will be handled by dataOwnerManagers which working with dataOwners. The dataOwnerManagers responsibilities are creating the new dataOwners and related dataSubOwners

Problem: In one time dataConsumers can process data of two (for example) the same NEW (should be added to the system before processing) dataOwners with same or different dataSubOwners for example: {1, 1, status} or {1, 2, status} . In this case the two NewOwnerEvent will be created and dataOwnerManagers can start creating the same NEW dataOwner record. Same situation can be for dataSubOwners but here we have at least 1 minute delay for creation. So if the first event will be {1, 1, status}, the second event {1, 1, status2} will be after 1 minute and it almost impossible that those 2 events will be handled in one time in dataConsumers.

So I need to prevent the resending NewOwnerEvent related to the same dataOwner - if the dataConsumer1 already send the NewOwnerEvent related to NEW owner1, I must notify (add possibility to check that the same event already exists) all other dataConsumers, that the owner1 have already been added to the queue and will be created soon. So I need some shared list of already sent/processed NewOwnerEvent events/dataOwners

Possible implementation: I tried using Redis to solve this problem. I’m adding the key each time I'm pushing the NewOwnerEvent event, so all other data processors can check this key to understand was the event already pushed or not. But the problem here is that in one time different dataConsumers can take the message from the same owner {1, 1, status} and {1, 2, status} and try to push event at the same time, because the key in Redis won't be created yet(dataConsumer2 check faster then dataConsuer1 add the key in Redis). So I need distributed lock to be sure that the Redis key will be added only by one dataConsumers at one time. I know that Redis already have implementation for distributed lock but I still confusing and guess it will be very complex solution in my case.


I'm "developing the distributed system" because the requirements are availability and scalability. I think I can't provide more details here. I tried to use abstraction for better understanding the common situation.

I can add here that the dataConsumers is part of Azure EventHub. Azure EventHub supports up to 32 partitions. The dataConsumer is the consumer of separate partition. As I said earlier “Data order is non-deterministic”. So, since I'm using EventHub I can't simplify at least the left part on my scheme.

dataConsumer is console app and just process (calibrate) and store the data flow.

The dataOwnerManager is Web API manages new data owners in system and all related settings. I decided that it will be separate bounded context. But I'm not sure about the dataConsumer.

Finally I decided that the dataConsumer can be part of the same context that dataOwnerManager and use the same DB storage. So I can remove the message queue in my scheme and just create shared service \ library that will create the new data owners in both dataConsumers (if necessary) and in dataOwnerManager through UI. Or just copy the code in both places.

But since the dataConsumers can handle the data from the same dataOwners in parallel. I still can get the situation when dataConsumer1 and dataConsumer2 will try to add the same NEW dataOwner in my DB.

One of the possible solution here is using unique key for dataOwner assigned Id(or use this assigned Id as primary key instead of surrogate key). Then I can handle the exception "dataOwner with this Id already exists" and forget it. Maybe it will be the best solution in my case. So, the NEW dataOwner will be added and all other attempts will be rejected on DB level I just need to handle it and the distributed locks not required.

But maybe somebody knows a better solution in this case. I still have the question can I do it in a better way?

  • 6
    The first four words of your post have me a bit skeptical. It may seem subtle, but the very mindset that you are "developing a distributed system" as opposed to "an application that provides some value in X domain" may be the biggest problem here. What if you designed this particular feature of your application in a way that is not distributed? Problem solved. If this is an academic pursuit, that is your system is distributed because you "want it that way", then please provide a more succinct, concrete example of your problem. Your dataOwnerN-1ConsumerNew3 jargon is difficult to follow. – king-side-slide Aug 8 '19 at 17:08
  • It had me skeptical too. – nadir Aug 9 '19 at 15:32
  • @king-side-slide, please check my update – Alexander Bikkuzhin Aug 9 '19 at 17:04
  • It's kind of unclear what the actual problem is, but it sounds like you have sequence dependent events (in which case normal error handling/retry should be fine) or non-idempotent events (in which case, you should fix that). Yes? No? (And in general, why is this all so complex?) – Telastyn Aug 9 '19 at 17:29
  • @Telastyn, I have queue of events, queue devided on partitions. Each partition has consumer. The event go to the first avaliable partition and will be handeled by the related consumer. It works fine on my case and adds scalability. But as I said event has owner in system. If I get the event wich owner is not in the system, I must add this owner to DB. So, 2 concurent consumer can add the same owner at one time. I can detect that the owner is new only into consumer. And then I have 2 ways, add it in consumer, or put in other queue and add later. I decided to handle DB exception(see my upd) – Alexander Bikkuzhin Aug 10 '19 at 6:07

As about your concerns with Redis (the lock is created before the actual Redis entry was created), I have encountered the same problem in my application.

What I ended up doing is using Redis Lua scripts, which are considered as a single atomic operation. I wrote a Lua script that tries to get the cached entry, if it doesn't exist, I try to acquire a lock on that Redis key (by trying to create a new cached entry named {myKey}_lock ). If lock acquisition also failed then the Lua script returns null, which indicates both actions has failed.

I just gave you an example, my point here is that you should take a look on Redis Lua scripts and see how they can help you achieve your required solutions while relying on Redis as your persistent store.

See more info here https://redis.io/commands/eval

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.