Best practice for multi-application communication and events?

Question

Essentially, I have a web application and a console application. The web application allows users to configure what will be processed, which is just a big information processor. From the web UI, a user creates a configuration that is added to a queue of configurations for processing.

When the user saves a configuration, if there isn't an instance of the console application running, the server will run the console applications (there will more often than not be multiple instances of the application started up for a particular configuration). In theory, instead of having a user enter a new configuration to start up an instance of the console app once the previous configuration is done processing, I need to pick up the next configuration in the queue.

The caveat being that the console application shouldn't have knowledge of its outer implementation. It just needs to be passed its part of the configuration and processes the information accordingly. Since there'll be multiple instances working independently on the same overall configuration, I can't pick up a new configuration until all of the instances of the application for the one in progress are finished.

My thoughts on this are potentially using a message bus for inter-application communication (RabbitMQ, MassTransit, etc). So once all of the processors are finished, I want to send out an event to notify that it's finished, so it will start up instances of the console application for the next configuration in the queue.

I'm not sure if a message bus is the best way to go about this or not, or if there are other solutions that I haven't thought of/come across. Worst comes to worst I can hack together something in the console application, but I'd like it to be as self-contained as possible.

Edit: for the time being, I'm just generating a request from the console application that hits an endpoint in my MVC project. I'm not thrilled about the idea, but it'll do for now I suppose.

Answer 1

I'm not sure if a message bus is the best way to go about this or not

Queues are the way to go, but you probably got the message direction wrong.

Ideal architecture for background workers is to have them provisioned (for example, a set of 4 workers) and let them consume what they have to do from a message queue. The reason it is so is because it handles well tasks of varying length, tend to make continuous usage of multi cores, and naturally handles if the consumers happen to lag down behind the producers. Tasks can optionally send data back once they are done, through a request or some other mean, but this is not required by the execution flow.

On your architecture, you spawn the workers directly from the web app with the data they have to handle. This causes you synchronization problems, because you have to manage the buffering manually, sync with workers dying to spawn new ones and so on. Additionally, you're not always optimally allowing resources.

Switching to a more traditional approach is likely to necessitate some refactoring but is definitely going to add more flexibility to your model. Later on, you may for example execute the workers on distinct machines for little cost. You also gain from using a queue implementation that usually ships with low footprint, persistence and fault tolerance features.