We have a .NET application where we need to scale a background job worker IHostedService to perform a long running task of compiling/building code requested by front end clients. Each task will receive a task Id so that front end clients can interrogate the status of the task. The intention is to use Docker containers for each of the background tasks. Each code compilation job would be on a separate container, so if one goes down, it wouldn't affect others.

We were planning to have another “supervisor” container do the scaling and keep track of all instances. We would maybe have a number of worker containers available for jobs but would increase the pool depending on the number of jobs coming into the system. How best to manually scale the worker containers? How do I communicate with a specific instance by task Id? It appears I need some sort of sticky session as front end client requests need to directed to the correct container.

  • I'm not sure I understand your question, but sounds like you need some message broker (e.g. RabbitMQ). Then you can add/remove workers based on the pending tasks queue size or some other metric. Also, for keeping and serving metadata about tasks you probably need to keep the data in some database. – freakish Apr 20 at 8:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.