I was wanting to create a program consisting of 3 elements:

  • A central component with no UI which will work on HTTP traffic and trigger one or more “worker” elements as needed
  • A monitor component which is an optional frontend to the central component which will read logs and send data to a central component through a mailslot (unless there is a good reason to change this)
  • Worker elements that will perform the batch processing.

To explain my choices so far, to give you an idea of where I am coming from (although I am open to suggestions). The central component will always be running although to keep it lightweight and silent. The monitor will only be launched when needed to check on the central component and to send commands to it. I have chosen the mailslot approach as it is very little data and infrequently sent on an ad-hoc basis.

I was wanting to keep the workers as separate processes, as they will need to process the data in parallel. Due to the data they are processing potentially having a short lifetime and some of the data is bulky which will slow an individual worker.

I was wanting thoughts on a few items below and although I appreciate there may not be a “correct” solution any considerations would also be welcome:

  1. I would like to run the central component as a service as it will keep it more hidden and allow it to run without having to log in. Although in terms of service vs process:
    • Are there security concerns with sending and receiving HTTP traffic to a service?
    • Are there any issues with services creating threads?
  2. Designing the workers, I like the idea of having these running in parallel although my question now is in terms of having 1 process per worker vs having 1 process with multiple threads:
    • Would there be any reason to have separate instances of the same processes running instead of running threads?
    • In terms of sending data from the workers to the central component what are the advantages of pipes vs mailslots, this data will be very small in size although it will be regular and frequent (~ once a second each worker will send a very small amount of data to the central component) and are there any better methods I have overlooked?

I can update the above with more information about any sections that are needed although I have tried to keep it brief. My true goal behind this project is learning about making different components communicate.

  • please don't cross-post: stackoverflow.com/questions/49260306/… "Cross-posting is frowned upon as it leads to fragmented answers splattered all over the network..." – gnat Mar 14 '18 at 14:17
  • @gnat Hi, I was in the process of deleting the question off the other site when you posted this. Please check again. – user2502611 Mar 14 '18 at 14:26
  • 3
    Have you considered message queues such as RabbitMQ? You'd get everything you ask for, minus some headaches (cross-platform, inter-process, language agnostic, reliable, with handy high-level semantics, etc.) for quite a small price (learning curve, one more component). That's information people willing to answer might need. – user44761 Mar 14 '18 at 15:28
  • @tibo Ideally I'd prefer to build something myself as this project is mostly to do with understanding how the communications work. Although I will consider this, Does Rabbit communicate over the network or locally? And how is it in terms or security and speed? – user2502611 Mar 14 '18 at 16:29
  • I understand your point - glad you precised it. Fwiw RabbitMQ is like a database server: you connect via sockets (using the AMQP protocol). Instead of writing to a table you publish to a queue (i.e. insert an element in a list). Clients can "subscribe" to a queue - i.e. get push notifications when something is written to the queue they listen to. Security is sufficient for 99.9% of all needs. And speed is mostly irrelevant (you get semantics instead, such as "this message has been delivered"). If you want an in-process queue to manage micro-tasks with threads it's certainly not the way to go. – user44761 Mar 14 '18 at 20:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.