My organization has a redundant setup across 2 data-centers. During normal operations both data-centers are up and handling workloads. For whatever reason, the infrastucture team has decided to operate 2 independent Kubernetes clusters, one in each DC.
We've just started developing a cloud-native application that runs as a sharded cluster and is deployed on Kubernetes. This application reads from an event stream and writes to a DB and more event streams.
Our plan is to deploy two application clusters (one per DC), and keep them running in parallel, in active-active mode, where both clusters are actively reading everything from the event stream and writing to the DB simultaneously. They both do all the work, and the first one to write it's output wins.
While this costs more hardware resources because everything is performed twice (once per DC), it also has some important advantages (zero downtime on DC failure, smoothing out transient latency spikes from one DC, no split brain issues across DCs).
So here are the questions:
Does this kind of architecture with competing application instances have a name?
We plan to write to a Postgre DB. Is there a way to perform this kind of highly concurrent updates where we expect everthing to be written twice from different transactions, without getting too many errors/rollbacks? Would MERGE statements, or ON CONFLICT clauses work?