1

We are currently rewriting our infrastructure from a monolith to a distributed system because the old system didn`t fit the workload anymore.

In one part of our distributed system, we have an Entity (from a database perspective) that has N ... events associated. We have a service S that handles access to all Ns and the events. The events get pushed by other Services. If no event has been pushed to a specific N for a specific time, S adds another event to N.

We want to be able to scale S horizontally (when we need it - It's more some kind of future-proofing). But there is a problem: If we run multiple instances of S on the same database, two or more instances might detect that there hasn't been an entry to one N for some seconds and add an event, both at the same time.

We came up with different approaches to solve this problem:

  • Lock the database, check if a new event should be added, add it and unlock the database. Might not scale well, because we have to stop all interactions with the table?
  • Implement S as master and slaves and let only the master do these updates. Might not scale - And electing a new master when the old master isn`t available anymore is a lot of work.
  • Shard S. Say one instance, S1, takes care of N1 to N500, S2 takes care of N501 to N1000 etc. But we need to add logic to take care of the sharding and resharding, if an instance goes down.

These options are either hard to implement or may not scale well (?). Is there a general solution to such a problem? Where multiple services look on the same dataset but must prevent each other from acting on the same things?

  • Regarding database locks: any DB access might involve locking. However, lock granularity would usually be a single row, not a whole table. E.g. I know that PostgreSQL supports explicit row locks. Consider investigating further whether DB locks could scale to your workload, as it would be the simplest of your suggested alternatives – the other two are basically reinventing distributed databases. – amon Jul 21 '18 at 16:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.