We have a .NET application where we need to scale a background job worker
IHostedService to perform a long running task of compiling/building code requested by front end clients. Each task will receive a task Id so that front end clients can interrogate the status of the task. The intention is to use Docker containers for each of the background tasks. Each code compilation job would be on a separate container, so if one goes down, it wouldn't affect others.
We were planning to have another “supervisor” container do the scaling and keep track of all instances. We would maybe have a number of worker containers available for jobs but would increase the pool depending on the number of jobs coming into the system. How best to manually scale the worker containers? How do I communicate with a specific instance by task Id? It appears I need some sort of sticky session as front end client requests need to directed to the correct container.