best practice when unit testing for embedded development

Question

I am looking for some best practice strategies for unit testing code written for embedded system. By embedded system, I mean code such as device drivers, ISR handlers etc., stuff that are pretty close to the metal.

Most of the unit tests are not possible without testing it on the hardware with the aid of a ICE. Sometimes, the embedded unit also needs to be hooked up to other stimulus such as a mechanical switches, stepper motors and light bulbs. This usually occurs in a manual fashion, automation would be great but hard and expensive to achieve.

Update

I came across a C testing framework that seems to be quite successful in testing embedded projects. It uses the ideas of mocking hardware. Check out Unity, CMock, and possibly Ceedling.

Update 06Jul2016

Came across cmocka - seems to be more actively worked on.

Answer 1

I would abstract away from the hardware dependencies at the earliest possible step, and build the system on software emulation/test harnesses, enabling all sorts of test frameworks. Often my developement PC was used to test as much as 95% or more of the complete system. The cost of the extra overhead (another layer of abstraction) was easily won back by the cleaner code generated as a result of that abstraction.

The testing of the truely baremetal parts of an embedded system is usually a seperate application (Unit test?) that hammers the firmware well beyond what the applications can even hope to achieve. Automation can be done - at a cost, but is not typical.

Unless, that is, you have the budget to build a unit test hardware harness including full ICE. This is absolutly fine as generally the functional tests are small.

Answer 2

A necessary tool to develop is a signal injector. The embedded system will have some way of interfacing with a host system (typically via a serial port reserved for debugging). Use this to send test data (best option is terse ascii formatted so it is easily simulated by humans too).

I completely disagree with this part of your question: "automation would be great but hard and expensive to achieve."

Using TeraTerm as a serial port signal injector, and writing some TeraTerm macros (takes about 20 minutes), there is a huge suite of automated tests which can be run against any part of an embedded system -- whether driver layer, O/S, layer 4-5, etc. TeraTerm: http://en.sourceforge.jp/projects/ttssh2/

If serial port is not available on the embedded system, then use a hardware tool to convert USB/serial port data to digital signals (also inexpensive and easy to achieve). As you read this, I am using a $30 microcontroller board (UBW: http://www.schmalzhaus.com/UBW32/ ) to test an embedded system for production, by injecting stimulus via TeraTerm macros which is sent via USB/serial to the microcontroller, which is running modified firmware which exercises digital inputs and monitors digital outputs of the target embedded system. In conjunction with this, we developed a python script (uses pyserial and pexpect) to automate the data injection and data validation. None of it is hard and none of it is expensive. In my experience, the managers spend the big bucks (such as $30,000 test equipment) when the test team is inexperienced and can't conceive of these easy solutions -- unfortunately, the general purpose big-iron equipment often does not include the test cases which catch the worst-case timing/etc of the target system. So the inexpensive method is preferrable for test coverage. Believe it or not.

Answer 3

This is a very difficult problem.

I have actually designed a unit testing harness for an embedded system, which would allow simulating hardware events/interrupts, and control the timing of the execution (to ensure we cover all the possible interleavings due to the concurrency), and it took a team of programmers more than 2 years to actually implement it and put it to work. That project is a proprietary development, but a similar (simpler in design) project is available here.

So yes, automation would be great. Yes, it is very hard and expensive to achieve. Yes, sometimes you have to do that. Rarely though, in my experience in most cases its faster and cheaper to use the stepper motors and light bulbs and make it all work manually.

Answer 4

Edit : my answer is close to mattnz's, I think...

---

I want to relate this problem with others, all tests that depend on something external to your code (like the system clock, a persistent filesystem or a database, contacting an external web service ...). I suggest the same policy for all of them, isolate the two levels in two layers of code.

Testing a single external operation

You might want to physically test each operation. Check that the system clock gives the correct time, check a file actually remembers what has been written, check a device receives a single operation...

These tests:

• should be as simple as possible: no algorithm whatsoever, no condition or loop
• might be order-dependant and machine-dependant: so you need to follow a strict order, and repeat on each hardware
• are mostly stable over the course of your project, so you don't need to run them that often
• so running them manually is an option ; automation is even better, if not overly complex
• Note that what is being tested is not your code, it is a tool that your code needs... So testing this might be optional for you, it might have been done by a different team...

Testing the logic (code, algorithm) that hooks together the external operations

By having a layer of code to make the actual external operations, by hiding them behing an interface that you can easily mock, your logic is not dependant on the actual physical devices anymore...

You can test simply, as any regular project, you are not in an embedded difficult-to-test code anymore.

Answer 5

Embedded CPU simulators can generally be programmed to also simulate hardware. All the virtualization technologies other than Xen do that. But you need to write code that pretends to have some registers at some physical address or, on x86, an address on the I/O bus, and then you need to respond to reads and writes to these addresses as if your software was a physical chip whose control and status registers were being accessed.

If you want to do this, I would suggest modifying QEMU. But it would not be easy. This sort of thing is generally only done when you are designing a custom chip with a microcontroller and some other cores for your I/O.

The development system sold by ARM Holdings provides for this and is likely easier to work with than hacking on QEMU, but is very expensive.

There are several Open Source ARM emulators that run a single subroutine, which itself can call other subroutines, that you can use for debugging tuning the performance of subroutines that don't depend on hardware access. I used one of these to great success to optimize an AES encryptor for ARM7TDMI.

You could write a simple unit test harness in C or C++, link the class or subroutine under test to it, then run it in the simulator.

I've been pondering a similar problem for years, how to unit test Linux or Mac OS X kernel code. It should be possible, but I've never actually tried. One possibly is to build a full kernel rather than testing your code in isolation, with the unit test framework linked directly into your kernel. You would then fire off the unit tests from some kind of external interface.

Maybe it would be more productive to use a code coverage tool, then test your firmware as a complete package through its external interface. The coverage tool would find code paths that were not yet tested, so you could then add additional external tests in an attempt to get more coverage.

Answer 6

As with non-embedded TDD, mock objects are definitely your friend.

Keep the interface to your underlying hardware clean and simple so that everything above the lowest level can be mocked out and you will have a much easier time of it - if you design your embedded application with testability in mind then testing will always go much more smoothly.

Also, just because you might not be able to test on-line until quite late in the project doesn't mean that you shouldn't prepare a suite of on-line tests too.

These should (initially) only need to test the bits that couldn't be tested offline. Sure, it's not TDD (since you are creating the tests in advance) but your offline TDD development should give you a good idea of what your hardware interface needs to look like and thus what online tests you need to perform.

Also, if online development costs much more than offline development (as it does where I work) then it could save you lots of time online having a well understood set of tests to run through.

Answer 7

In embedded developing you often do boundary scans to verify the whole application (including hardware) works. Also see JTAG for in system debugging.

Testing pure software routines without link to the hardware can be done by a standard C Unit Test framework like Check. But beware of memory limitations (especially stackspace etc. on small devices). Know your contracts! You can also try to abstract the software routines from the hardware to gain a bigger test-coverage but this is usually costly in terms of performance on embedded devices like small PICs or AVRs. However, you can mock hardware ports to achieve a bigger coverage (and of course you can test that mock, too).

You can also try to use emulators for the chip or circuit simluators, but these kind of tools are expensive (especially in combination) and complicated.