Reduce duplicates in outbox pattern in event driven systems

Question

Trying to implement the outbox pattern for an event driven system. The outbox pattern in a nutshell is a way to ensure system events are sent to the event log/queue/bus at least once (using the term event bus loosely here):

a separate process that periodically checks the contents of the Outbox and processes the messages. After processing each message, the message should be marked as processed to avoid resending.

My concern is I want to reduce the chance of duplicates because:

it is possible that we will not be able to mark the message as processed due to communication error with Outbox.

In this case when connection with Outbox is recovered, the same message will be sent again.

The proposed solution is idempotency which is fine with me but does it make sense to at least improve the message relay and reduce chance of duplicates? Are there patterns that exist to improve the message relay? If none, I'm thinking of the following:

Use 2 tables:

• OutboxTable (stores messages you want to send to the event bus)
• PublisherTable (stores messages already sent to the event bus)

Pseudocode for worker that reads the OutboxTable 24/7:

loop through each message...

if item is currently processing (by other workers)
    - skip

if item is marked as "published" in the PublisherTable (meaning a previous worker was able to publish the event but wasn't able to "confirm" it to the OutboxTable for whatever reason)
    - mark item as "published" in OutboxTable
    - skip
    
if item is was never processed or was processed but wasn't published
    - mark item as "processing" in PublisherTable  (so other workers can skip it)
    - publish/send item to the event bus 
    - mark item as "published" in PublisherTable
    - mark item as "published" in OutboxTable

(Each ongoing process has a defined timeout that other workers use to skip or process the item).

What I gather here is that I essentially made my own mini message queue just to achieve this. But since there's no way to make an atomic transaction between a DB and a real message queue, this approach kind of make sense.

So my question is, are there patterns similar to what I'm trying to do?

Answer 1

I have never seen any pattern or variation that wasn't just a shell game.

Fundamentally, you have two writes that you are trying to commit

• An update to the event bus (really, an update to the durable store of the event bus)
• An history in your own durable store

Because these writes are fundamentally using two different locks, there's always going to be some risk that your process exits before completing the second unit of work.

Welcome to the laws of physics.

Now, if the durable store for the event bus is your durable store, then you may be able to avoid some kinds of problems, because the transaction semantics of a single lock are all or nothing.

But if the stores are separate, then you are reduced to figuring out how to spend money to make the network more reliable, how to spend money to make your processes more reliable, and so on.

Answer 2

does it make sense to at least improve the message relay and reduce chance of duplicates?

It might make sense to do this, depending on your priorities (requirements) and options.

First off, you may implement whatever shenanigans in your application code - nothing is going to change the fundamental fact the the call to the Event Bus might fail for some reason, nor the probability of that happening - the Event Bus simply does not depend on your application code. And as soon as you encounter an unexpected failure when calling the Event Bus, your only option is to either retry the call or to ignore the message altogether. Unfortunately, a retry is the very thing that introduces the risk of duplicates. Your proposed solution is good in that:

• it helps avoid multiple workers processing the same event;
• it ensures that once an event has been processed, it will not be processed again.

That would be my basic implementation, and I would not add anything else, aside from idempotency.

Generally, as long as there are multiple separate systems (the DB and the Event Bus in this example), you will always have a problem. The communication between them is never 100% reliable, nor are the systems themselves (exactly everything in this world can break down; I would agree that this has more to do with Physics than computing), and so you always end up solving some flavor / cousin of the Two General's Problem, which is fundamentally unsolvable.

Depending on the exact situation (namely, the capabilities of the Event Bus) some approximate solutions might be available, e.g. Two-Phase Commits.

However, this is where you really need to evaluate your priorities, because such solutions typically have both performance and maintenance costs and might simply not be worth the improvement. Which honestly I believe to be true in this specific scenario.

So coming back to the question:

does it make sense to at least improve the message relay and reduce chance of duplicates?

Well, if you implement idempotent operations, then my answer is a firm "no". Otherwise - still most likely not. If your messages have IDs, using them as idempotency keys should lead to a trivial solution.

---

Also, the PublisherTable does not add anything of value - it's enough to have the OutboxTable.

for each message:
  if message is "processing" AND message timeout is not expired:
    /* the message is being handled by another worker */
    /* for those who don't know - this is the competing consumer pattern */
    - skip

  if message is "published":
    /* the message was already processed successfully */
    - skip
        
  else:
    /* the message is either new or is "processing" but the timeout expired */
    - mark message as "processing"
    - publish/send item to the event bus 
    - mark message as "published"