I have an Messaging Server Application that is responsible for brokering/proxying calls made from an Application Tier to numerous external services. The purpose of this application is to abstract the technicalities of invoking these services away from the main Application Tier.
This works well as the Application Server does not have to worry about http/ldap etc protocols, wcf, ftp, soap, ebxml etc etc. They simply send a "payload", with a few identifiers, and the Messaging Server handles the rest. It also means that if a service definition changes the Application Server does not need to be changed. Additionally the Messaging Server is back ended by an SQL 2008 database that stores an audit of all messages sent and associated responses etc.
The general data flow architecture of this is as follows:
Application Server(s) => Load Balancer => Messaging Server(s) => [X] => External Services
I need to implement a retry mechanism into the Messaging Application tier. The intention is to gracefully recover from situations where the Messaging Server isn't able to forward onto the target service (i.e. service down, network issues, timeouts etc), i.e issues with point [X] in the architecture above.
The high level design requirement is:
Application Server sends a request to the Messaging Server. This then attempts to forward to the external services. If the first attempt at this fails the Messaging Server responds synchronously to the Application Server stating that the message is "in retry"
The Messaging Server then proceeds to retry the fowarding per the contract (i.e. X retries with Y seconds between each).
One of two things will happen next, all contracted retries will have been performed unsuccessfully or one of the retries will succeed. In either case a message is sent back into the Application Tier to notify the state of the messaging request.
A few gothcas
The message to retry can't be held in "memory" as if that Messaging Server goes down the message is lost. Additionally a retry contract could be 5 times once every 12 hours, holding data in memory for that length of time isn't feasible. That said some retry contracts could be 5 times once every 5 seconds.
If the forward network goes down and then recovers the load of the retries should be spread across the entire messaging tier rather than a single server.
The commucation between Application and Messaging tier isn't a concern as that framework is already in place. However the architecture of the retry framework in the Messaging Tier is still up in the air. How would you implement this?
Options we have/are considering
On failure store the retry data in a database, then have a polling service that checks the database every second. If a message is found that is scheduled for retry this is push back onto the Messaging Tier via the Load Balancer
On failure store the retry data in a database, use a CLR job to poll the database and push messages, scheduled for retry, back onto the Load Balancer
May or may not be relevant:
- All code is C#
- Databases are SQL 2008
- Comms from Application to Messaging tier is performed via WCF with BasicHttpBinding.
- We have complete control over all aspects of the Messaging Server tier and no control over the Application Tier.
- The Messaging Tier currently handles around 500k transactions per hour, so you can imagine how quickly things will backup if there is a failure on one of the external services