Managing multi-server monthly transaction aggregations

Question

Context: A system tracks some sort of transactions (e.g. money flow) for it's whole user base. At the end of the month each entity capable of receiving transactions has to be sent exactly one bill containing all the transaction's of the current month. This system is instanced, meaning that there are multiple instances (servers) of the system running.

How to handle this sort of scenario? I must not allow duplicate monthly transaction evaluations. Given the nature of the system a master-slave architecture is ruled out as master-election or master-slave in general is not scaling to a large amount of instances.

I would like to spread the load of the monthly transaction aggregation across all running instances. The transactions are stored in a database in the form (user_id, transaction_target_id, transaction_amount). Failed monthly transaction aggregations should be retried by another instance

In short: I need exactly-once guarantees for a distributed system without a master-slave concept

Answer 1

Exactly-once guarantees can be had the easiest with a (option 1) message queue system. This also neatly solves the problem of retries, as failed operations can be stuffed back into the queue, or a dead letter queue can be used.

You can technically run n agents on the problem (option 2), and each agent only processes users with ((int)userId) % n == agentId). Or a less naive partitioning algorithm. But you'll still have most of the issues present in a consensus algorithm.

You can use RAFT directly (option 3)

You can extend the database table with an "in processing by node id" column (option 4). The nodes can then claim rows and use the presence of their value as a precondition for further operations.

I'd probably run an upstream duplication checking algorithm so duplicates can be caught before they get sent to a customer.