6

We are using Apache Camel with ActiveMQ in a microservice architecture in a growing system of integration solutions. I was wondering how each service should react when they are unable to connect to the message queue, or if this should be handled at all.

As the system is right now each service logs an exception for each reconnect attempt if the message queue broker is unreachable. This seems very sub-optimal. Are there any already implemented solutions or other smart suggestions that can handle this? We run 1 non-clustered broker on localhost, so if the broker is uncreachable, it is most likely caused by the service containing the broker actually having crashed and would not be up until someone manually restarts the entire servlet container (jetty in this case), which would most likely take minutes at least. And by the time someone restarts it the log files would have countless JMS connection exceptions.

  • Are those crashes indicative of quality or design issues? It seems reasonable to focus on the crashes first, then the restart issue, and finally the delayed message delivery or re-attempt issue. – rwong Sep 12 '16 at 1:20
  • Are you concerned about logs filling up/obscuring other messages? Or how the services should behave in this event? The comment from rwong seems the most sensible approach – matt freake Sep 12 '16 at 6:38
  • You may need some separation between Development, Test and Production environments so that crashes due to recently changed and unstable code do not affect production systems. – rwong Sep 12 '16 at 7:50
  • It might interest Upstart. Even if you fix the thing with these crashes, this sort of tools makes a big deal better the availability of the services. – Laiv Nov 10 '16 at 22:59
1

Java EE containers and Spring Framework manage JMS connections, as examples. My experience with them has been that they log an error and attempt to reconnect 3 or 5 seconds later. Since there is a delay between reconnect attempts, the number of log entries shouldn't be ridiculous for a reasonable amount of time.

0

There are many variables here. Ideally, one would need a greater understanding of the time constraints and the nature of the services you provide.

But in the general case I would like to advise the following: First, you should consider the probability of the different error cases. Don't add logic to handle an error which is unlikely to happen, because every addition also adds new error cases. You already have a local broker which (if I understand it correctly) will buffer messages in case the remote system goes down. This sounds like a decent principle which might save you from some network glitches. If you foresee these glitches, great. If not, I would skip the broker.

If you have the broker, it already sounds like your backup plan, so I wouldn't try to continue gracefully if it goes down. Try to make it as reliable as possible instead, finding a messaging system which is robust. You might want to have a timeout long enough to allow a quick reboot of the broker in case it crashes. But a startup time of several minutes makes this sound almost impossible, so you might want to reconsider your solution to starting the broker.

If you have a system with high load, buffering (with a local broker or not) will never save you in the long run. So you might want to reconsider and push error upstream instead. This approach has been promoted lately by the reactive manifesto and is more or less required not to bog your system down during temporary glitches. Return error codes when your system is in a failure state and force the clients to retry their requests.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.