0

My organization has a redundant setup across 2 data-centers. During normal operations both data-centers are up and handling workloads. For whatever reason, the infrastucture team has decided to operate 2 independent Kubernetes clusters, one in each DC.

We've just started developing a cloud-native application that runs as a sharded cluster and is deployed on Kubernetes. This application reads from an event stream and writes to a DB and more event streams.

Our plan is to deploy two application clusters (one per DC), and keep them running in parallel, in active-active mode, where both clusters are actively reading everything from the event stream and writing to the DB simultaneously. They both do all the work, and the first one to write it's output wins.

While this costs more hardware resources because everything is performed twice (once per DC), it also has some important advantages (zero downtime on DC failure, smoothing out transient latency spikes from one DC, no split brain issues across DCs).

So here are the questions:

Does this kind of architecture with competing application instances have a name?

We plan to write to a Postgre DB. Is there a way to perform this kind of highly concurrent updates where we expect everthing to be written twice from different transactions, without getting too many errors/rollbacks? Would MERGE statements, or ON CONFLICT clauses work?

6
  • Any chance of posting a diagram for this setup?
    – WMRamadan
    Sep 14 at 11:59
  • "active-active" is the name. You already used it yourself.
    – Useless
    Sep 14 at 14:24
  • For some reason, I've always thought of active-active as being some kind of load-balanced setup, where each of the active instances perform only a part of the work. I guess active-active does describe the fact that both clusters are simultaneously up, but is there a term to describe the fact that both instances are performing all of the work? That would help me out in finding further literature about this setup. Sep 14 at 15:41
  • N.B. you still have split-brain issues! You always have split-brain issues. What if the DCs lose connection to each other and then receive different, conflicting event streams?
    – user253751
    Sep 14 at 19:08
  • There is a single event stream cluster in this case. So any split-brain issue would be within that cluster, and is out of the scope of the question. Past experience has demonstrated that we are more likely to face split brain issues in our own apps than in established software components. Sep 14 at 19:17

1 Answer 1

0

To answer the Postgres part of the question, assuming that you are entering records with a well defined primary key, using something like ON CONFLICT DO NOTHING would work well, as it would simply ignore the conflicting record.

Edit: To clarify, by well defined I really mean something that will be the same no matter which cluster it is inserted from.

Just be more careful if the record uses some kind of randomly generated key which is created by the service or something, which is somewhat common in high throughput event streams, as obviously this will not work the same way and you may need to implement more logic to handle the conflict and this will make the process slower.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.