5

We have a set of micro-services all built according to the Domain Driven Design (DDD). The micro-services communicate via Domain Events with each other (OrderSubmittedEvent, CustomerBilledEvent, ...). We implemented everything with Spring Boot and use JMS with ActiveMQ as the message broker. All events are published to a topic and every micro-service can implement a JMS listener for an event of interest.

But for error handling, we heavily rely on ActiveMQ. As I said, all events are sent to a topic. However, we configured ActiveMQ to use Virtual Topics for the listeners, which means that ActiveMQ will create a separate queue for each listener and copy the message from the topic to the queue. If a listener fails, the message is sent to a Dead Letter Queue (DLQ). This helps us, because we can monitor the DLQ, see errors, fix them and put the message back to the queue of the listener which has failed for another try.

We have a couple of problems with that:

  • We end up with a lot of queues as each and every listener will get its own queue.
  • We cannot switch the message broker as we rely on ActiveMQ's Virtual Topics.
  • We cannot use ActiveMQ out-of-the box as we must configure it always. Our client for which we develop the software does the configuration manually in his system. As we have so many queues and little pitfalls, it's very cumbersome for him.

Well, we don't have any better idea, that's why I'm reaching out to you. How can we handle failing messages? What's your way?

Improve this question
1
  • I use polling and each consumer keeps track of what it consumed.
    – Constantin Galbenu
    Dec 27, 2017 at 16:43

1 Answer 1

Reset to default
5

One way to deal with this is to use pull-based listeners instead of push-based. Each listener keeps track of its last read message and can request "messages since X". You could use either polling or a notification when a new event comes in or both to trigger message requests. No more queues, but the listener will need to store its last processed message id. You also need to store the events in a way that preserves ordering to support listener requests.

Whenever a listener fails to process an event, it can log the failure to whatever logging infrastructure you have setup. Then you will observe or be notified of the failure by your monitoring infrastructure. Once you fix that particular listener and redeploy it, the listener will start back where it left off.

Improve this answer
3
  • I like the idea, but it also sounds like a lot of work. Every listener must be come a poller (= scheduled job) which holds some data (= database table). Is there any framework support or did you implement it all on your self?
    – Thomas Uhrig
    Dec 28, 2017 at 7:26
  • @ThomasUhrig Yeah, it is a lot of work. Many microservices proponents say that each service should have its own database anyway. Kafka can support pull subscriptions but hard to ops. So maybe look at EventStore which does pull subscriptions or write something custom for Postgres using LISTEN/NOTIFY. Microservices is not a great fit for low ops cost scenarios.
    – Kasey Speakman
    Dec 28, 2017 at 18:37
  • I spend my day playing around with a database table for my events and scheduled tasks which keep polling this table. It looks promising. Thanks for your answer.
    – Thomas Uhrig
    Dec 28, 2017 at 20:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.