Should we be using pub/sub in our messaging stack?

Question

Basically this question is of the variety "Can I check that we're not about to do something dumb?"

We're setting up a new system that needs to guarantee delivery of emails.

We have an existing system that processes financial transactions and we want to decouple that from sending receipts (which it is currently handling itself) so we're setting up a second service to offload sending receipts. The email service is completely independent, has it's own database, just receives events from the financial system to tell it what emails to send.

Initially we thought to use a pub/sub (probably Google's) for communications between the systems.

But as we designed the systems around it, we realised the worker process in the email system is simply taking messages off pub/sub and putting them into a Postgres table, which another process then works its way through sending emails from.

We've got buffering of messages to deliver at the publish side too, since even though google pub/sub is very reliable, it might not be 100%.

So we wondered why we even need the pub/sub, why not just have a webhooks system instead and have the messaging worker in the financial system ping a webhook on the email system?

Given that pub/sub is such a common design pattern for solving this kind of problem, I'm concerned that there's something we're missing in our thinking?

Performance seems like the obvious reason, but given that the financial system has to do the following:

• Put a transaction in it's database
• Contact the payment gateway
• Mark the transaction as paid
• Queue sending a message in the DB

(Then a worker unqueues the message and hits the webhook)

While the webhooks process in the email service only has to

• Put the message event in the database

  (let a worker process take care of processing that event)

It seems to me that the financial system necessarily will never be able to move fast enough to overload the webhook at the email service. Processing the transactions should always be a bigger bottleneck.

Answer 1

I have written a similar system about a month ago, and just like you I've discovered that this is best solved by a reliable queue technology with a redrive.

While some might argue using SQL as your queue is not the most reliable way, it definitely works in many cases.

The reason to use pub-sub systems is when you have multiple different things that need to happen from one event. E.g. if you had two different emails sent from one transaction, you may then have a system where

|====|    |====| => [UserEmail Sub]
|Tran| => |=Pub|
|====|    |====| => [ReceiptEmail Sub]

But even then, those little Sub components would just queue a different set of data onto your Queue mechanism, which ultimately would have multiple failovered worker processes pulling datasets from and sending actual emails.