Choosing right technology for messaging system

Question

I'm working on a messaging system for two years now, the system was written by a long ago gone team and involves emails and document processing. The basic process is:

1. Receive an email, parse it, save attachments to samba share. Notify message processor. We have a robust Java application doing this very well.

2. Process the message, this involves getting user data from LDAP, calling file processing web-services on different platforms. The system has a sort of re-delivery service that polls database for failed or timed out messages in different states and resends them.

The file processing parts are EJB web-services that basically run command line utilities. The problem lies in our orchestration solution which is based on OpenESB (almost dead now), it has a bunch of BPELs calling each other and calling remote EJBs (EJB3.0 on glassfish v2).

The biggest problem is that too much logic is done in BPELs (say, all database updates are done in BPELs, there's no persistance layer), they look like the most horrifying spaghetti I've ever seen. Not to mention they make NetBeans which we're using to edit them run really slow, to the point that some files are only editable in a simple text editor as XML. Moreover the system has a number of nasty bugs that would require major refactoring to fix and I'm dreaded to even think about it. These messages are handled manually by our support stuff so there's almost no client-impact but I would like to have a real transactional system anyway.

Now, for the question itself. I'm willing to spend a lot of my own free time trying to reach two goals: build a replacement for ESB and BPELs, learn new and preferably trendy technologies. I would like to keep the code in file processing EJBs since they run just fine, although I'm thinking of getting rid of SOAP as their remote interface. So I'm asking for any insight of which technology(-ies) would let me create a robust messaging solution that would:

1. Have a friendly persistence layer, there won't be anything really complex in dbs, just message meta-data.
2. Take care of balancing and polling calls to file processing web-services.
3. Wouldn't require dealing with tons of XML in different places in order to add a new interface method.
4. Would be scalable - run on a number of machines.
5. Would allow realtime monitoring, like viewing message queues, current load, statuses etc.
6. Hopefully will let me learn something new, meaning not J2EE stack. Basically I'm open to anything except BPEL and OpenESB.

Answer 1

I'm having a lot of fun with Erlang right now. It's a great language in and of itself. The real fun is in the framework and runtime it sits on. The Erlang Runtime is designed for distributed, concurrent applications (it was born at Ericsson for their network infrastructure). For integration with other systems, there is RabbitMQ which is built in Erlang and has native APIs for most of the big languages. It has a built in management interface that allows you to view messages and the like.

Addressing your requirements specifically:

1. Erlang has a built in database called Mnesia. For durable messaging, Rabbit leverages Mnesia (if an instance goes down the queue won't go down with it) and you don't have to worry about it yourself it just works.

2. Out of the box Rabbit uses round robin pub-sub. If you have 3 listeners to what's called a fanout queue they will be alternated between for each message.

3. Rabbit has an API driven configuration with UI's available to manage it without changing programs.

4. Check: Erlang was built with scalability in mind...it was originally designed to support Ericsson's network infrastructure. Combined with RabbitMQ and you can distribute your message processing across many nodes

5. Check: Rabbit MQ has many built-in and 3rd party plugins for Monitoring

6. Erlang is gaining a lot of traction as a solution for concurrent processing definitely a different approach outside of the standard OO paradigm.

Answer 2

For the remote interface I'd recommend considering RESTful instead of SOAP. I myself had bad experiences with SOAP in the past (particularly regarding performance and interoperability), and found REST a great replacement: lightweight and flexible.

For the ESB infrastructure your needs require a framework, to support the features mentioned it'd be burdensome to do otherwise. I don't have much experience on it, but one of my customers runs a quite complex ESB using Mule, with good results.

An alternative to ESB would be to use just an MQ framework. For instance, OpenMQ