How to handle errors best addressed in a higher abstraction layer and dependent on state inferred in current layer?

Question

I have a use-case that can be repaired, but the logic for repairing it is best done in a higher level of abstraction. Just throwing exception/failure is ugly because it's hard to pass back up through the stack the information needed for the parent-layer to know how to fix it.

To explain the the specifics, I have a program that controls multiple pieces of hardware. The 'top' layer is responsible for calculating how to task each piece of hardware to work together, how to generate 'paths' of data across hardware etc. My next layer includes threads for each individual device. each device thread generates and caches device state (fetched from hardware and config files). Devices process requests from the application, figure out how to update the hardware to make those requests happen, and then will modify the cached state to reflect the changes made. Finally I have plugins which send physical requests to hardware.

This works fine usually, requests are sequential to avoid data race issues. We only pass information down layers, we have no way of getting async updates back up the layer. However, it's very unlikely, but possible, that someone could physically change the hardware without going through our control system. We will wait until the plugin detects an error trying to handle a request, and only then try updating our state for that piece of hardware to see if a change caused the failure (constant monitoring is a later feature).

My problem is that when a change is detected on one device, that usually means a change was made to connected devices. When I detect this, I want to trace the change to connected device, update their state etc until I found all changes (this doesn't require updating all devices, just connected ones). Each device thread is only responsible for one device, so they can't trace the change through other devices without propagating the exception up a layer. However, I need to know exactly what changes were detected on the device to know which connected devices I should be refreshing the state of. If I just throw an exception, this knowledge is lost and must be recalculated, which is a small but real cost due to time it takes to communicate with hardware.

The tempting thing to do is to call a method in my application layer saying "this change was detected, go update everything else related to it". However, this makes drivers aware of the layer above them, which is not good. Another option is to simply propagate the error up and let my application layer notice the error and then go back to the same device to ask what changed and update all it's state, then re-calculate the changes to be made and make the request back down to the device level. The first is an obvious code smell, the second seems like more code, redundant logic, and small speed penalties (it takes a little time to get state from hardware). Are there some other options?

Answer 1

This sounds like a job for the observer pattern. Every device is being observed by it's connected devices. Are you sure you need the parent layer to get involved?

Unless there is something else you need the parent layer to do then just handle these exceptions on the device by recovering, setting it as having changed, and looping through it's connected observers telling them what they need to know to recover.

If one hop recovery is enough you're done. If you need to allow the recover to propagate some fixed n hops you'll need some stopping logic or it'll cascade to everything. You already said you didn't need that.

After all that throw an exception to politely tell the parent layer that the request died because some goofball has been mucking with the hardware. Log that you've recovered from their shenanigans and are ready for the next request.

You set all that up by making sure whenever there is a connection between two devices the two devices register as observers of each other.