8

I am at the beginning of a project which involves reading from several sensors and fusing the data from those sensors together. Altogether there will be 4 sensors connected via USB and a webcam, also connected via USB.

One of my colleagues is very vocal about how nice it is to split up programs into smaller parts and have them communicate over the network. He suggests that we should have an executable for each sensor (or camera) and then a central controlling application which communicates with the others.

I intuitively don't like this idea. The colleague in question worked on another project which used that approach and had no end of problems that were hard to track down and debug.

It doesn't seem like a very state-ful design and strikes me as somewhat inelegant. I would like to write a library for dealing with each sensor and maybe run them in separate threads.

It should also be pointed out that the calculations we need to do will provide updates to another system at nearly 1000Hz. Adding in a layer of network communications seems like adding a potential bottleneck.

I would be interested to hear other peoples' opinions on this and perhaps some references regarding this type of practice.

  • 9
    "had no end of problems that were hard to track down and debug" not sure that a multi-threaded solution will be easier to debug. Not saying that multi-threaded is wrong, Just don't ever recall hearing "Let's use a mutli-threaded solution, that is so much easier to debug." – cdkMoose Sep 1 '15 at 16:02
  • 3
    As a frequent user of Erlang, this "use the network/protocol as the basic layer of abstraction" is often my default mode of thinking. It does make things easier to debug (you can test complete behaviors in isolation) and it makes a system more robust (camera #1 putting something in a weird state can only crash camera #1's handling code, and nothing else waits on that code), and makes adding a supervision system to it simple. These traits are very difficult to achieve otherwise -- but then again your problem might really be trivial enough that none of this matters. – zxq9 Sep 1 '15 at 19:13
  • 1
    How are you running 1000Hz on USB in the first place? What exactly are your performance requirements? Do you consider the sockets to become a latency or a throughput bottleneck? – Bergi Sep 1 '15 at 19:28
  • 1
    Why do you think you need a "stateful design"? Aren't your sensors event-based? – Bergi Sep 1 '15 at 19:29
  • A programmer had a problem. He thought to himself, "I know, I'll solve it with threads!". has Now problems. two he – Toby Speight Nov 9 '16 at 10:26
13

I intuitively don't like this idea.

Well, intuitively, I like the idea of splitting up programs into smaller parts. But if the different processes run always all on the same machine, network communication is probably not the optimal form of IPC. For highspeed communication between processes, shared memory might be the better option. But whatever approach you choose, you have to measure or at least estimate the performance before making any judgement calls.

The colleague in question worked on another project which used that approach and had no end of problems that were hard to track down and debug.

You have to check what kind of problems, and where the root causes were. If they had problems because of concurrency, you will pretty much encounter the same problems (if not more) when you try a multi-threaded solution.

provide updates to another system at nearly 1000Hz

If that is "high speed", depends on the speed of the production machines, and the size of operation involved in each update. When it comes to performance, gut feelings are extremely unreliable, you need to measure things. If your colleague believes his approach will be fast enough, and you believe it will not, at least one of you will have to proof or falsify his believe. For example, one of you could implement a small prototype to simulate the interprocess communication, and measure the speed. Everything else is looking into a glass bowl.

9

They may or may not be relevant to your application, but some benefits to the multi-process solution you haven't seemed to consider are:

  • Finer-grained permission control. You can have separate permissions for the webcam and the other sensors.
  • Easier to test sensors in isolation.
  • Easier to mock out a sensor at runtime.
  • Easier to get third parties to write code for their sensors without having to open your code.
  • Use of tools like wireshark for troubleshooting, both during development and in the field.
  • The only context I'm aware of where "stateful" is seen as a positive attribute is when the stateful parts of a system are restricted to a small scope like an actor, which seems to support your colleague's design more than yours.
  • Easier to release a lock and restart just one sensor's process without having to restart the entire system.
  • Can scale just by adding hardware.
  • Socket communication with source and dest on the same machine is relatively efficient.

Additional drawbacks to the multi-process solution:

  • Dealing with backpressure is more complex.
  • If you're essentially just passing on the raw data from each sensor without any filtering or processing, you've just changed your controller application from reading from a usb driver to reading from a network driver, without any real gain in abstraction.
  • If one uses an architecture where each socket has a master side and a slave side and each transmission by the master generates an immediate response by the slave (even if the response is "not ready yet"), and the master always waits for the slave's response, would backpressure be a problem? – supercat Sep 1 '15 at 19:10
  • That would be one way to handle it, @supercat. It's not that backpressure is unmanageable, it's that a lot of people fail to account for it up front, making the design look simpler than it really needs to be. – Karl Bielefeldt Sep 1 '15 at 19:21
8

The approach your colleague is advocating is often referred to as a microservices architecture and is quite in vogue these days. It has a number of potential advantages:

  • Scalability: it can make it relatively easy to scale by simply adding extra machines / VM instances.
  • Robustness: with an appropriate design it can be made tolerant to individual microservice processes crashing, losing connectivity or otherwise becoming unavailable.
  • Extensibility: using standard Web technologies to interface between microservices (typically http REST requests with Json or XML responses) can make it relatively easy for third parties or customers to interface with your system and extend it for their own needs.

It also has a number of disadvantages:

  • Performance: generally you will pay a performance cost for the inter-process communication. This may be very significant if you build http REST style APIs.
  • Complexity: typically you will need quite a lot of extra code to marshall data between processes vs. just passing data around between threads in a single process. You will also introduce dependencies on libraries for network communication, http support, etc.
  • Debuggability: it is typically more complex to debug an application made up of multiple independent communicating processes vs. using an IDE's integrated debugger on a single process.

I'm not sure quite what you mean when you say the design "doesn't seem like a very state-ful design". Generally "state-ful" is considered a bad characteristic of a design and "stateless" microservices are considered good design.

Given the information on your requirements you provide in your post however, I think a microservices based design would be overkill for your use case and the benefits would not justify the additional complexity. The benefits of microservices architectures tend to only really come into play when building larger scale web services that place a premium on scalability, robustness and extensibility.

The potential benefits of a microservices architecture are only realized with a well thought out design as well. In my opinion, such an architecture requires more up-front design (particularly when it comes to the protocol for communicating between microservices) than a single process design if it is to be successful at solving the problem at hand.

  • 1
    "typically you will need quite a lot of extra code to marshall data between processes vs. just passing data around between threads in a single process" -- there's a trade-off here though, between the amount of code you write in order to pass messages, vs the design simplicity that you gain from (e.g.) communicating sequential processes. If you get the CSP design "right" then it needn't necessarily make any difference whether the components are separate threads in the same process, or separate processes, since either way they're using the same message interface with different implementations. – Steve Jessop Sep 1 '15 at 19:13
  • To add on to what @SteveJessop is saying about serialization/marshalling -- this can be made very efficient later once the performance characteristics of the system are understood. The common case of "uh, networks/sockets are hard, let's use HTTP and XML" is nearly the worst case, and can be vastly improved upon -- but is familiar enough to most of us that its a great way to hack a system together that works right now. Tweaking something that works (and is therefore already measurable) is much nicer than trying to troubleshoot a design that does not yet exist in practice. – zxq9 Sep 1 '15 at 19:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.