Hardware test GUI with multiple event sources (4 serial ports and user interaction) - best architecture?

Question

I have the following requirements:

1. Write a GUI app for automated testing of some custom PCB hardware.
2. The system must be usable for a technician with little to no advanced PC skills.
3. The test setup has 4 different serial port interfaces:
   • DUT (Device Under Test) USB-C port. (Used for debug and sending test commands)
   • DUT RS232 port. (used for communication to PLC in final system.)
   • Testjig RX USB-C port. (Port on identical hardware to the DUT, used to test RF network.)
   • Testjig USB port. (Connected to STM32 Nucleo dev board, used to measure various voltages.)
4. The tests are frequently of the form: send a command to device A, wait for answer from device B, check if answer is correct.

My issue is that there is an existing codebase which does not work reliably. Less experienced me struggled to get the System.IO.Ports serial port class to work, so it has all sorts of weird attempts at workarounds and generally just ugly spaghetti code, blocking the event loop with Thread.Sleep() calls, etc.

I found later that it wasn't just me who struggled with Microsoft's serial port class, so I switched to the SerialPortStream library. However, the codebase is currently extremely brittle and I absolutely hate to have to change anything in it. I want to use the opportunity to try to structure the code better and clean up as much as possible.

My current idea is to try to create a kind of "event driven" architecture in which each serial port listen for data which would denote a full message in its DataReceived event handler, and then enqueue a full message in its queue automatically. It should also Invoke a delegate to pop messages from the queue each time it appends a message. This function should parse each message and check for messages it recognizes.

The system will become a giant event-driven state machine which has nested state machines for each event source (user, DUT USB, DUT RS232, testjig RX, testjig Nucleo) as I see it.

My question: is this the best way to go about it? Are there easier/better options available? And if I go with this giant state machine, how should I protect my different threads spawned by different event handlers from data races when they have to read/update the shared state?

Answer 1

I did something very similar for a company I used to work with, a system for testing and commissioning motor-generator controllers.

Platform for the test runner was .NET (WPF), the testing procedure was minutes long with hundreds of checks, which pulled me towards using Reactive Extensions and I ended up modeling the test process as a series of commands (test step parameters, easily (de)serializable to and from YAML/JSON) and corresponding command handlers (the actual steps that orchestrate I/O abstractions - I/O interface with DUT, external CLIs for MCU flash tools etc.). Command handlers emitted messages into a reactive stream that various receivers could listen to and update GUI, send them to an actual server to be saved and analyzed, etc. Message itself could be either be a dumb log (output from said CLI for example), or assertion that can halt the test and flag nonconformal units if they fail.

An option would be to go all in on reactivity and model the I/O abstractions (and test procedure itself) as actors (Akka or ProtoActor in .NET) or actor-like entities (e.g. hardware abstractions running as separate processes, communicating via stdin/out or message brokers). Actors are pretty good at handling isolation of state (hardware components are likely stateful and isolated in the real world as well), well suited for expressing state machines, and you can use supervision strategies to handle functional safety (in our case, power output to all hardware was to be shut down on procedure failure or breach of containment zone of in-circuit tester, leading to reset of the testing device and reinitializion to default state).

Of course, this is all based on some presumptions that were valid for my case, such as acceptable latency between individual steps, scope and scale of the system. Some questions to ask yourself:

• do you want to monitor and react to transients?
• do you have any hard real-time constraints?
• are you testing 5 devices or hundreds of different types?

Personal anecdote: These kind of systems are its own unique version of integration hell, and asynchronous message-driven systems are notoriously hard to debug. Take caution with drawing the initial consistency boundaries, and take your time with modeling the events and event flows that will connect individual components.