Watchdog/recovery mechanism for realtime embedded system (using heartbeat, exceptions and Posix signals)?

Question

We have a large(ish) real-time embedded system. It's VxWorks, if that makes any difference. It has some C code in DKMs, but is 95%+ in C++.

It has absolutely no exception handling, nor Posix signal handling. It is multi-threaded, so if one thread/process/subsystem dereferences a null pointer, or access a non-existent vector entry, etc, I guess that it just dies(?) and the rest of the system ... limps along without it?

We need a robust mechanism to handle such anomalies and keep the system running smoothly, unattenuated, without human intervention.

Here are my first, somewhat jumbled, thoughts, and I would appreciate any comments on whether this is too simple, too complex, missing something, etc. I am sure that this is industry standard and that there are accepted best engineering practises. What are they?

For the C code, I plan to add a signal handler to catch segmentation faults, etc. It would be too much effort to add meaningful exception handling to the C++ code, so I had thought a single try/catch around the main() function. However, while those can log & swallow "a bad thing happened", I am not certain that they can identify the offending software and "make it better", and it seems a bit heavy-handed to restart everything, rather than just the offender.

Perhaps (the above combined with) a watchdog or heartbeat mechanism?

A watchdog in main() could know the process Id of each thread, since it started them, and periodically check their status, killing and restarting any which are hanging or have died.

Or a heartbeat mechanism, where the main() periodically sends a message to each thread and start a timer. If the timer expires before an ACK is received, kill & restart the thread (I use the term thread loosely; they might be processes).

The above sounds sort of vague, but is perhaps a reasonable start. What is a good design, preferably one used often in similar circumstances?

---

[Update] After investigation, I find that our C++ code builds on a framework, which we cannot alter, and which runs everything as threads, nor processes.

My latest proposal is that we:

• handle errors in the C code by signal handler (whether one or one per DKM, if these are processes)
• add a single try/catch around the main function
  • both of the above will log the problem to a disk file with as much diagnostic information as possible and "somehow" restart all of the software (as opposed to individual threads)
• when the system starts up, it will launch a watchdog process. This process will launch the application as a single process and monitor it at regular intervals. If it cannot be found, or is "stuck", then it will be (killed and) restarted.

I do not see much point in complicating the watchdog process, which must be robust and should never fail. To that end, I am not sure of the advisability of a heartbeat or handshake keep-alive mechanism. If we, then I would imagine a single interchange between the watchdog process and the application process, but that would only be meaningful if the the application process had a similar monitor/handshake mechanism with each of its threads.

I will put that to the powers that be and see just how much robustness they wants and how much effort they are willing to invest. If it helps anyone to comment, this is a DAL D project, so not overly critical.

Answer 1

VxWorks documentation uses the word "task" to refer to what traditional systems call a "thread". I will refer to the concept as "thread" here, given the audience.

failure domains

You repeatedly confuse "thread" and "process" in your description.

A process has one or more threads of execution, all sharing the same address space. If any of its threads corrupts memory with Undefined Behavior like a null dereference, then the whole process should exit. And in your use case, it sounds like you want to immediately restart it.

You don't have a choice in this. Once a thread within a process fails, the entire process fails, including all of its threads.

So we have failure domains at the granularity of processes. It makes sense to restart a failed process.

I'm willing to believe this bullet point from wikipedia:

• User-mode applications ("Real-Time Processes", or RTP) isolated from other user-mode applications as well as the kernel via memory protection mechanisms

So processes are protected from one another, similar to traditional systems. When I say "traditional", think of Linux, FreeBSD, MacOS, Windows. Non-RTOS systems with the usual notions of processes and threads.

VxWorks configures the MMU so that a given process is simply unable to change the memory of another process. It cannot corrupt another process in that way.

The various threads of a process, OTOH, share memory and can cause corruption visible to peer threads, which is why we restart the whole process after an error.

I am ignoring the < 5% C code in DKMs that you mentioned, since those loadable kernel modules have been given high privilege and can do a great deal of harm if they're buggy. Failure in a DKM should typically imply a reboot of the host. Don't deploy buggy DKM code.

nannies

It sounds like one or more of your apps exhibit occasional buggy behavior which leads to the app crashing and exiting. If you usually launch app1, but want immediate restarts after such an exit, just launch app1_nanny.sh instead:

#! bash

while true
do
    app1
    echo Restarting at $(date)
done

It's a very simple nanny script that keeps an eye on that wayward app.

I will usually put a brief sleep 1 before the echo, to avoid burning CPU cycles needlessly if the app manages to get into a mode where it happens to fail immediately upon startup.

watchdog timer

Some buggy apps will hang instead of exiting. Asking the app to log a heartbeat message every K seconds is a good idea.

Have a very simple companion process read the log or otherwise look for those heartbeats. When 2 × K seconds have elapsed without a peep, it should send a kill signal to the buggy app, making it drop out.

The idea here is that the companion process is so simple that it can't show buggy behavior, it is always available. It's the same rationale for the nanny parent process which spawns a buggy child application process.

log analysis

It sounds like you have a complex system, with multiple components, which you do not yet understand. You want to fix that. You will need to identify the Root Cause of an observed failure if you ever hope to bug report it and obtain a software fix.

Crank up the log level of components you suspect of being buggy. Send logs to an analysis host, which can filter "routine, boring" messages. Each time there's an observed failure, assign a staff member to write a report of what was observed, what the suspected Root Cause is, and what bug report fixes you're awaiting. You need a process to track the details, so you can systematically eliminate one Root Cause after another.

simplify

Your host is running too many things.

Buy another host. Run a subset of those things. Measure MTBF and contrast the two systems. Does the simpler setup fail less often? Which app(s) seem to be the culprit? Is system load relevant? Is there some "kiss of death" network input that always crashes an application?

You have lots of debugging work ahead of you.

Answer 2

if one thread/process/subsystem dereferences a null pointer, or access a non-existent vector entry, etc, I guess that it just dies(?) and the rest of the system ... limps along without it?

A null pointer failure inside a thread will require to end the whole process to where the thread belongs to (with all it's other threads). Fullstop. There is a high risk the adress space of the process is corrupted, which will affect not only the thread where the crash occured.

If other processes (not threads!) in your system are affected depends on how they interacted with the crashed process, and in case they did, how tolerant they are written. Maybe they are not affected because they don't communicate with the other processes directly. Maybe they can handle not-running other processes gracely. Failure tolerance in regards to other processes can be implemented with or without exception handling or signal handling, it does not really matter.

A watchdog in main() could know the process Id of each thread, since it started them, and periodically check their status, killing and restarting any which are hanging or have died.

It seems you are confusing process IDs and threads here, so let me say this more clearly: only processes guarantee the necessary isolation to keep a system alive after a null pointer crash, not threads.

The idea of a watchdog or heart beat monitor is fine, but it needs to be implemented as a separate process, not in the main routine of an application which can potentially crash. The monitor process should be a very simple and small program, so simple that it has a very low risk of crashing itself.

I use the term thread loosely; they might be processes

That's the problem: you should use the terms more strictly (and I am wondering actually why you don't - it seems you are talking about an existing system, so most decisions about threads and processes should have already been made).

Threads are lightweight and can easily communicate with each other through shared variables in memory - that's their benefit over processes. On the other hand, they don't provide the address space isolation you need to recover from a crash - that's their disadvantage.

This terminology is used today on most major platforms. I made a short Google search and from what I found, VxWorks is not really different from that (the docs call processes Real Time Processes, and threads Tasks - with a hint that thread and task are synonyms).

Answer 3

if one thread/process/subsystem dereferences a null pointer, or access a non-existent vector entry, etc, I guess that it just dies(?) and the rest of the system ... limps along without it?

No.

This means that the thread has invoked undefined behaviour, at which point your entire system is suspect - the thread can happily of overwritten memory being used by other threads, etc. You should restart the entire application, not just the offending thread.

Answer 4

One thing to remember in addition to everything else: Make sure that a process cannot get into a state where it stops making progress, but still manages to keep a watchdog timer alive.