We process messages through a variety of services (one message will touch probably 9 services before it's done, each doing a specific IO-related function). Right now we have a combination of the worst-case (XML data contract serialization) and best-case (in-memory MSMQ) for performance.
The nature of the message means that our serialized data ends up about 12-15 kilobytes, and we process about 4 million messages per week. Persistent messages in MSMQ were too slow for us, and as the data grows we are feeling the pressure from MSMQ's memory-mapped files. The server is at 16GB of memory usage and growing, just for queueing. Performance also suffers when the memory usage is high, as the machine starts swapping. We're already doing the MSMQ self-cleanup behavior.
I feel like there's a part we're doing wrong here. I tried using RavenDB to persist the messages and just queueing an identifier, but the performance there was very slow (1000 messages per minute, at best). I'm not sure if that's a result of using the development version or what, but we definitely need a higher throughput[1]. The concept worked very well in theory but performance was not up to the task.
The usage pattern has one service acting as a router, which does all reads. The other services will attach information based on their 3rd party hook, and forward back to the router. Most objects are touched 9-12 times, although about 10% are forced to loop around in this system for awhile until the 3rd parties respond appropriately. The services right now account for this and have appropriate sleeping behaviors, as we utilize the priority field of the message for this reason.
So, my question, is what is an ideal stack for message passing between discrete-but-LAN'ed machines in a C#/Windows environment? I would normally start with BinaryFormatter instead of XML serialization, but that's a rabbit hole if a better way is to offload serialization to a document store. Hence, my question.
[1]: The nature of our business means the sooner we process messages, the more money we make. We've empirically proven that processing a message later in the week means we are less likely to make that money. While performance of "1000 per minute" sounds plenty fast, we really need that number upwards of 10k/minute. Just because I'm giving numbers in messages per week doesn't mean we have a whole week to process those messages.
=============== edit:
Additional information
Based on the comments, I'll add some clarification:
I'm not sure serialization is our bottleneck. I've benchmarked the application, and while serialization does show up in the heat graph, it's only responsible for maybe 2.5-3% of the service's CPU utilization.
I'm mostly concerned about the permanence of our messages and potential misuse of MSMQ. We are using non-transactional, non-persistent messages so we can keep queueing performance up, and I would really like to have at least persistent messages so they survive a reboot.
Adding more RAM is a stopgap measure. The machine has already gone from 4GB -> 16GB of RAM and it's getting harder and harder to take it down to continue adding more.
Because of the star routing pattern of the application, half the time an object is popped then pushed to a queue it doesn't change at all. This lends itself again (IMO) to storing it in some kind of key-value store elsewhere and simply passing message identifiers.
The star routing pattern is integral to the application and will not change. We can't application-centipede it because every piece along the way operates asynchronously (in a polling fashion) and we want to centralize the retry behavior in one place.
The application logic is written in C#, the objects are immutable POCOs, the target deployment environment is Windows Server 2012, and we are allowed to stand up additional machines if a particular piece of software is only supported in Linux.
My goals are to maintain current throughput while reducing memory footprint and increasing fault tolerance with a minimum outlay of capital.
