We are currently rewriting our infrastructure from a monolith to a distributed system because the old system didn`t fit the workload anymore.
In one part of our distributed system, we have an Entity (from a database perspective) that has
N ... events associated. We have a service
S that handles access to all
Ns and the events. The events get pushed by other Services. If no event has been pushed to a specific
N for a specific time,
S adds another event to
We want to be able to scale
S horizontally (when we need it - It's more some kind of future-proofing). But there is a problem: If we run multiple instances of
S on the same database, two or more instances might detect that there hasn't been an entry to one
N for some seconds and add an event, both at the same time.
We came up with different approaches to solve this problem:
- Lock the database, check if a new event should be added, add it and unlock the database. Might not scale well, because we have to stop all interactions with the table?
Sas master and slaves and let only the master do these updates. Might not scale - And electing a new master when the old master isn`t available anymore is a lot of work.
S. Say one instance,
S1, takes care of
N1 to N500,
S2takes care of
N501 to N1000etc. But we need to add logic to take care of the sharding and resharding, if an instance goes down.
These options are either hard to implement or may not scale well (?). Is there a general solution to such a problem? Where multiple services look on the same dataset but must prevent each other from acting on the same things?