0

Let's say I've a sendMessage endpoint. This endpoint looks at configured message destinations and then puts 1..N messages into a queue (e.g. {userId: 3, type: SMS, trackingID: X} and {userId: 3, type: EMAIL, trackingID: X}) and returns the trackingId.

Now I want to expose /status/x, which will display an overall status and sub-status:

{
id: X,
status: IN_PROGRESS,
tasks: [{type: SMS, status: DONE}, {type: SMS, status: IN_PROGRESS}]
}

My initial idea was to put an initial overall state and an initial task state for every task in a database. Then I would put each message into a queue. But what if the application crashes after putting only one message into the queue. Or what if the processor of the queue crashes all the time, hence never send a status update message? What if the status update message is lost? What if it arrives after 2 hours, you dont want that SMS anymore.

Can anyone point me to an architecture / book where this is covered in great detail? Lost status updates, super late messages, partial execution.

2
  • 2
    Eventual consistency, fallacies of distributed computing, ... this might take several books to answer.
    – Laiv
    Mar 3, 2023 at 13:49
  • yes, you basically want to read about distributed computing in general, I think. Fun fact: It's impossible to make sure an email is only sent once. Completely impossible by the laws of physics and math. That's because the internet might crash at the exact moment the email server says it got your email.
    – user253751
    Mar 3, 2023 at 18:35

2 Answers 2

1

But what if the application crashes after putting only one message into the queue. Or what if the processor of the queue crashes all the time, hence never send a status update message? What if the status update message is lost? What if it arrives after 2 hours, you dont want that SMS anymore.

The most common architectural pattern to address these concerns is event-driven architecture, where messages and events are produced and consumed by various components of the system. This architecture allows for loose coupling between components and enables fault tolerance and resilience.

In an event-driven architecture, each event is published to a message broker, and subscribers receive and process those events. You can use a message broker such as Apache Kafka, RabbitMQ, or Amazon SNS/SQS to manage the events.

More precisely, you should implement the following:

Idempotency: You can make sure that messages are processed only once by adding a unique identifier to each message and storing it in a database. Before processing each message, you can check if the message has already been processed by looking up the identifier in the database.

Error handling: You can implement retry and error handling mechanisms to handle situations where messages fail to be processed. For example, you can configure a maximum number of retries for failed messages or send failed messages to a dead-letter queue for further analysis.

Timeouts: You can set a timeout for each message to ensure that messages are processed within a certain timeframe. If a message takes too long to be processed, you can consider it failed and handle it accordingly.

Event sourcing: You can use event sourcing to store and replay events in case of failures. This ensures that the system can be rebuilt from a sequence of events, even if parts of the system fail or data is lost.

For more information on event-driven architecture, you can check out the following resources (my personal favorite is the last one):

  • "Designing Event-Driven Systems" by Ben Stopford
  • "Building Event-Driven Microservices" by Adam Bellemare
  • "Event-Driven Architecture Overview" by Martin Fowler
0

My initial idea was to put an initial overall state and an initial task state for every task in a database. Then I would put each message into a queue. But what if the application crashes after putting only one message into the queue.

For this particular question, the way you are asking implies that you are treating the initial request as 'fire-and-forget' which is a mistake, IMO. All asynchronous transactions should start with a synchronous request/response. That is, until your asychronous system has confirmed that it has received the request, you cannot assume the request is being handled. That doesn't mean it will get to its final destination without issue, just that your asynchronous system has taken responsibility for handling it and managing the status.

For the rest of it and the larger questions about what literature exists around these types of systems, I would start with Gregor Hohpe: https://www.enterpriseintegrationpatterns.com/gregor.html

For example, this seems relevant to your questions: https://www.enterpriseintegrationpatterns.com/patterns/messaging/GuaranteedMessaging.html

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.