Handling cases where the message queue is unreachable

Question

We are using Apache Camel with ActiveMQ in a microservice architecture in a growing system of integration solutions. I was wondering how each service should react when they are unable to connect to the message queue, or if this should be handled at all.

As the system is right now each service logs an exception for each reconnect attempt if the message queue broker is unreachable. This seems very sub-optimal. Are there any already implemented solutions or other smart suggestions that can handle this? We run 1 non-clustered broker on localhost, so if the broker is uncreachable, it is most likely caused by the service containing the broker actually having crashed and would not be up until someone manually restarts the entire servlet container (jetty in this case), which would most likely take minutes at least. And by the time someone restarts it the log files would have countless JMS connection exceptions.

Answer 1

Java EE containers and Spring Framework manage JMS connections, as examples. My experience with them has been that they log an error and attempt to reconnect 3 or 5 seconds later. Since there is a delay between reconnect attempts, the number of log entries shouldn't be ridiculous for a reasonable amount of time.

Answer 2

There are many variables here. Ideally, one would need a greater understanding of the time constraints and the nature of the services you provide.

But in the general case I would like to advise the following: First, you should consider the probability of the different error cases. Don't add logic to handle an error which is unlikely to happen, because every addition also adds new error cases. You already have a local broker which (if I understand it correctly) will buffer messages in case the remote system goes down. This sounds like a decent principle which might save you from some network glitches. If you foresee these glitches, great. If not, I would skip the broker.

If you have the broker, it already sounds like your backup plan, so I wouldn't try to continue gracefully if it goes down. Try to make it as reliable as possible instead, finding a messaging system which is robust. You might want to have a timeout long enough to allow a quick reboot of the broker in case it crashes. But a startup time of several minutes makes this sound almost impossible, so you might want to reconsider your solution to starting the broker.

If you have a system with high load, buffering (with a local broker or not) will never save you in the long run. So you might want to reconsider and push error upstream instead. This approach has been promoted lately by the reactive manifesto and is more or less required not to bog your system down during temporary glitches. Return error codes when your system is in a failure state and force the clients to retry their requests.