I was wanting to create a program consisting of 3 elements:
- A central component with no UI which will work on HTTP traffic and trigger one or more “worker” elements as needed
- A monitor component which is an optional frontend to the central component which will read logs and send data to a central component through a mailslot (unless there is a good reason to change this)
- Worker elements that will perform the batch processing.
To explain my choices so far, to give you an idea of where I am coming from (although I am open to suggestions). The central component will always be running although to keep it lightweight and silent. The monitor will only be launched when needed to check on the central component and to send commands to it. I have chosen the mailslot approach as it is very little data and infrequently sent on an ad-hoc basis.
I was wanting to keep the workers as separate processes, as they will need to process the data in parallel. Due to the data they are processing potentially having a short lifetime and some of the data is bulky which will slow an individual worker.
I was wanting thoughts on a few items below and although I appreciate there may not be a “correct” solution any considerations would also be welcome:
- I would like to run the central component as a service as it will keep it more hidden and allow it to run without having to log in. Although in terms of service vs process:
- Are there security concerns with sending and receiving HTTP traffic to a service?
- Are there any issues with services creating threads?
- Designing the workers, I like the idea of having these running in parallel although my question now is in terms of having 1 process per worker vs having 1 process with multiple threads:
- Would there be any reason to have separate instances of the same processes running instead of running threads?
- In terms of sending data from the workers to the central component what are the advantages of pipes vs mailslots, this data will be very small in size although it will be regular and frequent (~ once a second each worker will send a very small amount of data to the central component) and are there any better methods I have overlooked?
I can update the above with more information about any sections that are needed although I have tried to keep it brief. My true goal behind this project is learning about making different components communicate.