We have a job-queue shaped hole in our architecture, but it isn't immediately obvious that there's a piece of existing software to fill it. There may be a reason for this, or there might not, so let me explain the problem.

We have jobs that need execution asynchronously. Some jobs may require all cores in the system for several hours, and many gigabytes of memory. Other jobs may require marginal memory, a single thread, and several milliseconds to execute, and have soft deadlines for completion, measured in single-digit seconds. And there are plenty of jobs in between, that take 1..100 minutes using all available cores.

We'd like to dispatch jobs with an approximate estimate of the resources required to execute it, along with some kind of priority or deadline. And we'd like 1..n machines to take jobs from this queue and execute them. But, of course, if a machine isn't big enough to run a job, it shouldn't accept the job; if it's currently running a job that takes half of available memory, it should only accept jobs that fit in the remaining space; etc. This is a scheduling / packing / constraint satisfaction problem that needs solving whenever the job queue changes.

From a lot of searching, there doesn't seem to be any open-source message queuing systems with constraint-based job scheduling. We can't be the only people who have a problem like this, and I'm wondering what other people do to solve it.

Note that we can't simply add lots of machines running job queue clients, and rely on excess capacity to solve the problem. We sell software as a service to the financial sector, and one of the security requirements is that their data is restricted to dedicated (virtual) machines, with separate subnets and cross-communication prohibited. If we allocate too many worker machines to a customer's network, our profits are directly affected by increased hardware rental costs, VM licensing costs, extra management complexity, etc.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.