I am looking for an advise on selecting or building a job placement algorithm.

In my company we have a simple computing platform built on top of kubernetes. Multiple clients, send compute jobs to a single Kafka topic, multiple workers continuously pull for new tasks from the queue, execute jobs and go back to pick up next tasks. Any worker can execute any job, the system has practically a single queue.

I need to modify this system to become data/cache aware. Imagine that each job requires some data [D1, D2, ... Dn], when this job lands on a worker [Wi] it first needs to retrieve this data and cache it locally before it can start execution. When a new job comes in with a requirement for some data [Dx], I want it to be assigned to a "worker pool" where workers already cached that data [Dx]

I am free to modify the architecture to implement the new functionality, for example we can replace Kafka or use multiple topics or introduce some look up tables etc.


  • The number of datasets is not known in advance, new datasets come at runtime.
  • Consider that worker can store/cache infinite number of datasets.
  • It has to be a push based mechanism where worker selects a queue/topic to get the next task from.

Perhaps an algorithm like this already exists. I would appreciate any direction. Thank you.

  • 2
    Is using a distributed cache an option? If all of the workers share a cache, it doesn't matter which worker you use. Just throwing it out there since I can't help with the actual question. Commented Jun 28 at 13:34
  • 1
    I think @IvovanderVeeken has the best approach. Use a distributed cache, and then your system doesn't need to care which worker processes the job. Commented Jun 28 at 13:57
  • 2
    A distributed cache is simple, but their are cases like the issue described where the data to be cached is large >GB and needed with fast low latency access, such that a distributed cache would require excessive bandwidth requirements. Commented Jun 28 at 14:14
  • 1
    The normal Kafka architecture would be multiple partitions per topic (so that you can have multiple consumers), and to give each message partition key. Here, you might want the data ID as the key. Unfortunately, you need multiple items of data (so can't have a single partition key), and it seems you don't really want to utilize partition-based concurrency. A better solution may be a shared cache (k8s volume) between the workers – storage is probably virtualized anyway, there's no latency difference between my cache and another worker's cache. You might want a network file system like Ceph.
    – amon
    Commented Jun 29 at 17:49
  • Thank you for all of your suggestions. I have considered a shared cache (and it might be the final solution if I will not find anything better), but it might not work well if you need to share large datasets (as mentioned above). Shared k8s volume will work well within the scope of a Node, but we have 100s of nodes, so this reduces the problem by the order of Pods per Node.
    – AG14
    Commented Jun 30 at 19:46


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.