We have a large(ish) real-time embedded system. It's VxWorks, if that makes any difference. It has some C code in DKMs, but is 95%+ in C++.

It has absolutely no exception handling, nor Posix signal handling. It is multi-threaded, so if one thread/process/subsystem dereferences a null pointer, or access a non-existent vector entry, etc, I guess that it just dies(?) and the rest of the system ... limps along without it?

We need a robust mechanism to handle such anomalies and keep the system running smoothly, unattenuated, without human intervention.

Here are my first, somewhat jumbled, thoughts, and I would appreciate any comments on whether this is too simple, too complex, missing something, etc. I am sure that this is industry standard and that there are accepted best engineering practises. What are they?

For the C code, I plan to add a signal handler to catch segmentation faults, etc. It would be too much effort to add meaningful exception handling to the C++ code, so I had thought a single try/catch around the main() function. However, while those can log & swallow "a bad thing happened", I am not certain that they can identify the offending software and "make it better", and it seems a bit heavy-handed to restart everything, rather than just the offender.

Perhaps (the above combined with) a watchdog or heartbeat mechanism?

A watchdog in main() could know the process Id of each thread, since it started them, and periodically check their status, killing and restarting any which are hanging or have died.

Or a heartbeat mechanism, where the main() periodically sends a message to each thread and start a timer. If the timer expires before an ACK is received, kill & restart the thread (I use the term thread loosely; they might be processes).

The above sounds sort of vague, but is perhaps a reasonable start. What is a good design, preferably one used often in similar circumstances?

[Update] After investigation, I find that our C++ code builds on a framework, which we cannot alter, and which runs everything as threads, nor processes.

My latest proposal is that we:

  • handle errors in the C code by signal handler (whether one or one per DKM, if these are processes)
  • add a single try/catch around the main function
    • both of the above will log the problem to a disk file with as much diagnostic information as possible and "somehow" restart all of the software (as opposed to individual threads)
  • when the system starts up, it will launch a watchdog process. This process will launch the application as a single process and monitor it at regular intervals. If it cannot be found, or is "stuck", then it will be (killed and) restarted.

I do not see much point in complicating the watchdog process, which must be robust and should never fail. To that end, I am not sure of the advisability of a heartbeat or handshake keep-alive mechanism. If we, then I would imagine a single interchange between the watchdog process and the application process, but that would only be meaningful if the the application process had a similar monitor/handshake mechanism with each of its threads.

I will put that to the powers that be and see just how much robustness they wants and how much effort they are willing to invest. If it helps anyone to comment, this is a DAL D project, so not overly critical.

  • 1
    I guess you do realize that a signal handler, to be safe, can do almost nothing? I.e. calling external code is dangerous. Commented Jan 5 at 22:31
  • If the goal is to have the app be restarted, and if there is a Watchdog process, what would you recommend? Could the app restart itself? Kill itself and let the watchdog spot that & restart it? Write to a disk file? We will settle for "if something goes wrong (terminate and ) restart the app". I hope that you can post a suggestion as an answer
    – Mawg
    Commented Jan 6 at 10:04
  • 1
    Kill the process and let the watchdog restart the process. You can write to or simply touch a file to update a timestamp, to see if the process is alive. Make sure you don’t start a duplicate. You could also use a shared object in the OS to signal. The other responders all have excellent advice. Read up a bit on your OS. Commented Jan 7 at 19:20

4 Answers 4


VxWorks documentation uses the word "task" to refer to what traditional systems call a "thread". I will refer to the concept as "thread" here, given the audience.

failure domains

You repeatedly confuse "thread" and "process" in your description.

A process has one or more threads of execution, all sharing the same address space. If any of its threads corrupts memory with Undefined Behavior like a null dereference, then the whole process should exit. And in your use case, it sounds like you want to immediately restart it.

You don't have a choice in this. Once a thread within a process fails, the entire process fails, including all of its threads.

So we have failure domains at the granularity of processes. It makes sense to restart a failed process.

I'm willing to believe this bullet point from wikipedia:

  • User-mode applications ("Real-Time Processes", or RTP) isolated from other user-mode applications as well as the kernel via memory protection mechanisms

So processes are protected from one another, similar to traditional systems. When I say "traditional", think of Linux, FreeBSD, MacOS, Windows. Non-RTOS systems with the usual notions of processes and threads.

VxWorks configures the MMU so that a given process is simply unable to change the memory of another process. It cannot corrupt another process in that way.

The various threads of a process, OTOH, share memory and can cause corruption visible to peer threads, which is why we restart the whole process after an error.

I am ignoring the < 5% C code in DKMs that you mentioned, since those loadable kernel modules have been given high privilege and can do a great deal of harm if they're buggy. Failure in a DKM should typically imply a reboot of the host. Don't deploy buggy DKM code.


It sounds like one or more of your apps exhibit occasional buggy behavior which leads to the app crashing and exiting. If you usually launch app1, but want immediate restarts after such an exit, just launch app1_nanny.sh instead:

#! bash

while true
    echo Restarting at $(date)

It's a very simple nanny script that keeps an eye on that wayward app.

I will usually put a brief sleep 1 before the echo, to avoid burning CPU cycles needlessly if the app manages to get into a mode where it happens to fail immediately upon startup.

watchdog timer

Some buggy apps will hang instead of exiting. Asking the app to log a heartbeat message every K seconds is a good idea.

Have a very simple companion process read the log or otherwise look for those heartbeats. When 2 × K seconds have elapsed without a peep, it should send a kill signal to the buggy app, making it drop out.

The idea here is that the companion process is so simple that it can't show buggy behavior, it is always available. It's the same rationale for the nanny parent process which spawns a buggy child application process.

log analysis

It sounds like you have a complex system, with multiple components, which you do not yet understand. You want to fix that. You will need to identify the Root Cause of an observed failure if you ever hope to bug report it and obtain a software fix.

Crank up the log level of components you suspect of being buggy. Send logs to an analysis host, which can filter "routine, boring" messages. Each time there's an observed failure, assign a staff member to write a report of what was observed, what the suspected Root Cause is, and what bug report fixes you're awaiting. You need a process to track the details, so you can systematically eliminate one Root Cause after another.


Your host is running too many things.

Buy another host. Run a subset of those things. Measure MTBF and contrast the two systems. Does the simpler setup fail less often? Which app(s) seem to be the culprit? Is system load relevant? Is there some "kiss of death" network input that always crashes an application?

You have lots of debugging work ahead of you.

  • Hi, thanks for some excellent feedback. I hope that I have clarified the process/thread thing in the updated question. Its a single process, with multiple threads. I think that I will add a second, "nanny", process as watchdog. We do, of course, have to cater for the DKMs, with one or ore signal handlers (tbd). Yes, we do log file analysis & bug fixing, but nothing is perfect, so we need a final "catcher in the rye" when deployed in the field (See what I did there? ;-) And, the more I look at it, the simpler it should be - any problems, log the problem, then (terminate and) restart the app.
    – Mawg
    Commented Jan 6 at 10:20
  • 1
    You have a single app1 process that is flaky. It likely has more than a Single Responsibility. Find some responsibility that you can rip out and assign to a new app2 process, so you have a pair of simpler processes. Monitor MTBF. Rip out app3 functionality if you can. // The DKMs must be rock solid. Evict flaky logic into an app process in order to simplify them. // A watchdog monitor process (which sends a kill) is distinct from a nanny parent (restarter) process. If you have observed "fail hang", you need watchdog to monitor keepalives. If only "fail crash", then there's no need.
    – J_H
    Commented Jan 6 at 16:46

if one thread/process/subsystem dereferences a null pointer, or access a non-existent vector entry, etc, I guess that it just dies(?) and the rest of the system ... limps along without it?

A null pointer failure inside a thread will require to end the whole process to where the thread belongs to (with all it's other threads). Fullstop. There is a high risk the adress space of the process is corrupted, which will affect not only the thread where the crash occured.

If other processes (not threads!) in your system are affected depends on how they interacted with the crashed process, and in case they did, how tolerant they are written. Maybe they are not affected because they don't communicate with the other processes directly. Maybe they can handle not-running other processes gracely. Failure tolerance in regards to other processes can be implemented with or without exception handling or signal handling, it does not really matter.

A watchdog in main() could know the process Id of each thread, since it started them, and periodically check their status, killing and restarting any which are hanging or have died.

It seems you are confusing process IDs and threads here, so let me say this more clearly: only processes guarantee the necessary isolation to keep a system alive after a null pointer crash, not threads.

The idea of a watchdog or heart beat monitor is fine, but it needs to be implemented as a separate process, not in the main routine of an application which can potentially crash. The monitor process should be a very simple and small program, so simple that it has a very low risk of crashing itself.

I use the term thread loosely; they might be processes

That's the problem: you should use the terms more strictly (and I am wondering actually why you don't - it seems you are talking about an existing system, so most decisions about threads and processes should have already been made).

Threads are lightweight and can easily communicate with each other through shared variables in memory - that's their benefit over processes. On the other hand, they don't provide the address space isolation you need to recover from a crash - that's their disadvantage.

This terminology is used today on most major platforms. I made a short Google search and from what I found, VxWorks is not really different from that (the docs call processes Real Time Processes, and threads Tasks - with a hint that thread and task are synonyms).

  • 1
    @MawgsaysreinstateMonica: I think before you create such a solution, you should have some statistics at hand how often it happens that the system crashes with a null pointer exception, and how often it happens that the system stalls / runs into an endless loop. Or maybe there are different crash scenarios like vector indexes out of range, or memory leaks? So as always, make sure you are solving the right problem - when "endless loops" happen once every 3 years, you probably don't need to implement a heartbeat, a simple watchdog which monitors the process would be enough.
    – Doc Brown
    Commented Jan 5 at 14:15
  • 1
    @MawgsaysreinstateMonica: so you have a large program already which was never run in a production-like test environment, so you cannot collect any stats about the reliability? I think for a system which is developed over months before it gets into the customer enviroment, that's is something you will really need, otherwise expect the project to fail, regardless how many "just-in-case" handlers you will add to it.
    – Doc Brown
    Commented Jan 6 at 10:56
  • 1
    @MawgsaysreinstateMonica: then I don't understand your comment. For a program which has been used for years, your organization should have some information available about the frequency and type of crashes. And when your time is sparse, it is even more important that you invest it in solving the right problem.
    – Doc Brown
    Commented Jan 8 at 13:46
  • 1
    @MawgsaysreinstateMonica "The bad news is that our C++ code builds on a framework, which we cannot alter, and which runs everything as threads, nor processes." Well, don't use the framework for your watchdog. Make it a separate process. You already (in the question) identified that the watchdog should be as simple as possible to reduce bug surface area. Frameworks move complexity around; but they actually increase the total complexity.
    – Ben Voigt
    Commented Jan 8 at 21:02
  • 1
    @BenVoigt: maybe there is a misunderstanding of what the OP was describing, but "not using the framework for the watchdog" is missing the point. The watchdog itself can be as simple as shown in J_Hs answer under "nannies", but that will still not allow to continue the monitored application process when a single thread crashes with a severe error.
    – Doc Brown
    Commented Jan 8 at 21:14

if one thread/process/subsystem dereferences a null pointer, or access a non-existent vector entry, etc, I guess that it just dies(?) and the rest of the system ... limps along without it?


This means that the thread has invoked undefined behaviour, at which point your entire system is suspect - the thread can happily of overwritten memory being used by other threads, etc. You should restart the entire application, not just the offending thread.

  • Thanks for that. It's threads all the way down, alas. Our C++ code builds on a framework, which we cannot alter, and which runs everything as threads, nor processes. I will add an update to the question, with a proposed solution and would very much welcome your comments on it.
    – Mawg
    Commented Jan 5 at 11:30

One thing to remember in addition to everything else: Make sure that a process cannot get into a state where it stops making progress, but still manages to keep a watchdog timer alive.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.