Assuming the following scenario:

A web application that receives inputs from the user and process the inputs through an algorithm which takes between 5 to 25 minutes (depending on the inputs) and provide a differed result. By the differed results I mean, the user won't wait for the result behind the UI and he will be notified through an email when the calculation is done.

  • The algorithm part which processes inputs should be scalable.
  • The application should be hosted on premises.
  • Requests came from paid users should be at the front of the queue.

I am trying to design a high-level architecture and am new to the world of scalable software, containers and microservices.
This is a rough basic design which I came with so far: enter image description here
In above:

  • The front-end app is responsible for receiving user inputs. (doesn't need to be scalable for the time being) and does publish requests into the messaging server.
  • The messaging server is a server that hosts a software like RabbitMQ.
  • The "queue manager" is a software that needs to be developed which is subscribed to the messages and allocates requests when an unallocated instance of the algorithm runner is available. Also, it is responsible for the order of the queue, depending on the plan which the user is subscribed too, so the paid user's request will be prioritised.
  • Algorithm runner instance is inside a container (for example docker) which we can scale up by increasing the number of the instances.

Here are my questions.

  • Does this architecture/design make sense at all? Is it not overkill or the other way around, it may be too simple?

  • My biggest doubt is, is there need for the queue manager app at all or should let the containers be subscribe to the messages directly. If that's the case how the prioritisation would work?

  • Can the prioritization be done by simply having multiple queues, and each queue would be processed based on priority? eg. Of Queue 1 is empty, go to Queue 2, if that is empty go to Queue 3, etc..
    – Euphoric
    Oct 19, 2018 at 11:17
  • @Euphoric It's an option, however, the business logic may get more complicated as in the request for user X should be processed at a certain time.
    – Shahin
    Oct 19, 2018 at 11:19
  • For the priority part I used rabbitmq.com/priority.html and it worked well for me.
    – Imdad
    Jan 13, 2021 at 6:57
  • I came here looking for what you called "queue manager". Let me know if it worked well for you?
    – Imdad
    Jan 13, 2021 at 6:59
  • 1
    @Imdad Yes, indeed. We ended up implementing this solution and it has been working quite well so far.
    – Shahin
    Jan 18, 2021 at 18:50

1 Answer 1


Only one comment - personally I'd keep the Queue Manager service.

While it might be possible for the "worker" containers to subscribe to the messages themselves to perform the algorithm tasks, it's be inefficient/difficult/impossible to implement several pieces of functionality (which IMHO would sooner or later be needed) from executing the code in a "worker" context:

  • providing a dynamic/global view of the system/service
  • performing the priority scheduling
  • performing the worker container monitoring and failure/recovery logic
  • bringing the containers up and down as needed/scaling the worker container pool
  • scaling up the entire service

With a dedicated, centralized Queue Manager it would be much easier IMHO to implement the above items:

  • it already has the centralized/global view of the service
  • it owns the centralized job scheduling decision, it's almost trivial to implement priority or any similar quality of service logic
  • it could be the central entity monitoring the status of the worker containers performing jobs, easy to detect failures. Could be pulled out as a separate microservice.
  • the service info used in making the scheduling decisions, complemented by service status info obtained from the scheduling activity are typically the very bits of info necessary for making working container pool scaling decisions. The Queue Manager can itself drive the execution of those decision or just provide triggers to a separate service dedicated for that.
  • scaling the system would not normally require scaling the worker containers (which would be required if they'd perform the queue manager functionality).
  • Thanks for the detailed answer. In my diagram, the Queue manager service is an independent application developed by myself and it's separated from the orchestrator which can be Kubernetes or Docker Swarm. Do you have the same view? Also, how would the communication between the Queue manager and workers occur? It is not clear in my head that how does the Queue manager aware of the existing workers in order to allocate jobs etc.
    – Shahin
    Oct 21, 2018 at 13:45
  • The orchestrator would be the one actually performing the worker pool scaling - not a separate service. The QM would just trigger its actions and gets the pool status. Oct 21, 2018 at 14:12
  • Since it does that it knows which workers it should have available, thus be able to assign individual jobs to individual workers. The assignments would be available in the global system view, the workers would get their assignments from there. You could also have them contact directly the QM to report status/pull instructions. Or you could make the workers servers 'listening' themselves for instructions pushed from the QM (but since the QM itself is a server you need to be extra careful as this leaves room for communication deadlocks). Oct 21, 2018 at 14:12

Not the answer you're looking for? Browse other questions tagged or ask your own question.