Given a workload with many long running CPU/IO intensive operations (e.g. producing multi-GB text files from database reads and business rules), what is the best way to balance .net application load across a group of Windows Server 2012 R2 VMs?
Ideally, the approach would allow for adding/removing/patching nodes as needed. It's pretty much a 24/7 operation with little opportunity for maintenance windows.
We are in the process of migrated millions of lines of COBOL code from a IBM mainframe to C#. The current workload is batch centered and there are lots of jobs that fire up at a scheduled time to perform units of work of various sizes.
Wherever possible, we'll look to make the converted processes more real-time, event-driven, etc... However, because of dependence on existing inputs/ouputs to/from many upstream and downstream partners and a desire to get from one platform to the other in the shortest time possible with minimal disruption, we're going to have constraints that prevent us going with a completely greenfield approach.
Due to the large volumes of constantly changing data involved and the interconnectedness of that data with other legacy systems, moving to a cloud-based infrastructure is probably not feasible.
Approaches we're considering
JMS queues - We have an enterprise class JMS compliant ESB and we could queue the work, having consumers on each VM that could pull work based on resource availability.
Task coordinator - We've considered creating a custom workload manager that would monitor servers and decide where to send work. Something along these lines.