How do I write unit tests for robots (and other mechanical devices)?

Question

I'm a member of my high school's robotics club, and am responsible for programming the robot. One suggestion I keep hearing from various adults is that I should be writing unit tests to help validate my code. The code base is getting a bit big, and I do agree that unit tests would be really helpful in helping me catch bugs quicker.

However, I'm not entirely sure how I could accomplish this. To the best of my knowledge, unit testing is done by taking a function (or a subsystem of the code) and feeding it a set of input to make sure it comes out with the same output each time. The code that I currently have doesn't do any heavy data crunching, but rather directly manipulates the hardware components on the robot. Most of the complexity comes from making sure that the electronics are sound, that the code at the moment matches the actual hardware on the robot, etc. Often times, I can only see if there's a problem by loading the code to the robot itself, and attempting to run it.

By extension, how can unit tests be written for code meant to operate any mechanical device? It seems to me that you can only catch errors by physically observing the operation of the machine.

Or am I just misunderstanding how unit tests should work?

(If it matters, here's the code, it's written in C++, and I'm participating in FRC)

Answer 1

You will need to mock the hardware layer to do this testing. The idea is that instead of your code talking to the actual hardware, you talk to a fake version of the hardware that you can control and then use to verify that your code is working correctly.

Unfortunately there are some problems you have to overcome:

• Mocking things in relatively low-level languages is more difficult (and thus, a lot more work)
• Mocking hardware-level stuff is more difficult (and thus, a lot more work)

Also, most of the value of automated, unit testing comes from being able to run your tests at any time to catch regression bugs for long periods of time after you write the original code. With these sorts of competitions, your code won't be used at all after the contest is over, so you won't really get most of the value out of the tests. And considering how difficult it would be to add tests to your particular case, it might be better use of your time to just do manual testing and focus on features for the contest.

Answer 2

I can think of a couple of things that you will need to consider. The first is to make your hardware access layer as thin as you can, even if that means creating a basic wrapper-type of layer for it. This offers you a couple of advantages. The first is that it allows you to isolate the hardware-specific behaviours of your code from the hardware access itself, which means you can test everything down to the very bottom of your hardware comms without needing to access the hardware itself.

For example, if you need to design an I2C-based signalling protocol, you can test your code is generating the correct I2C signals without needing to hook the hardware into your tests.

For calls to the actual hardware, you can test that they are behaving correctly by mocking your hardware layer, and this is where keeping a very thin hardware layer really pays off, because you can reduce your mock to needing only handle the minimum functions required to actually address the hardware, but you don't necessarily need to test the individual signals themselves, as all of your signalling should have been testable at a higher level. This means then that you use your mock to check that calls are made to specific hardware methods that cause your signals to be sent to hardware. If you need to poll your hardware, then your mock needs to be able to trigger events or methods in your code only, because again, your return signalling should be handled in a higher layer.

This basically fits with what Oleksi said in his answer, in that it is usually more work to mock hardware-level stuff, however it isn't so difficult if you keep to the leanest possible minimal-code/call layer you can make for the hardware.

When you have code that passes all of its tests, you will still need to run through a series of manual tests in order to be sure you hoked everything up correctly in your hardware layer.

The other thing that comes to mind aside from the mocking and the layering, is to use a test-first development practice. Essentially, you code your requirements as test criteria, and you base your implementation off your tests. This will help you to ensure you keep your implementation code to a minimum, while ensuring all of your test cases are driving your development efforts. By not wasting too much time on other potentially non-critical code that you might be tempted to do "just because", test first helps you to stay focused, and will make it easier to change your code as you debug, as will the use of your unit tests and mocks. Debugging software bugs through the hardware is notoriously complicated and sucks up large amounts of your time that you would find better spent on other tasks.

Answer 3

I can tell you how they do it on Flight Simulators

First, you're only going to get half the answer if you ask this question to only programmers so you should probably cross post this on http://electronics.stackexchange.com while you're at it.

I haven't done any work with robots but I did spend 5 years doing hardware on flight simulators so I can tell you how their architecture works.

The hardware layer is dumb

It contains a basic interface where you can adjust simple input/output values and set interpolation breakpoints for analog signals. When you're working with 'fresh' hardware, everything will work as expected with little or no calibration but over time the parts will undergo mechanical wear and need to be adjusted.

The calibration is a simple table that contains sectioned ranged between the min/max values. To measure input on these, a servo is typically used (ex a linear potentiometer, transducer, accelerometers, etc). Or in the case of instrumentation, you just judge accuracy visually and adjust until calibrated.

The software layer is the opposite

Everything is complex and interconnected so it's important to isolate some variables to test the functionality. There's no need to give yourself a headache thinking up scenarios because it's much easier to run some realistic scenarios where you can collect data. When you run the tests, you're basically measuring the stored data against the current output.

On a flight simulator this is referred to as a QTG (Qualification Test Guide). At its core it plots the data on a 2D grid where one dimension is time and the other is the output.

Believe it or not, that's the essence of how they develop the models. A real plane is equipped with a ton of sensors and flown around doing controlled scenarios. Because all of the controls can be driven without human interaction, the tests are run (ie the sim flies itself) by the computer and the data is compared.

Even though robotics are created on a much different scale the principles are the same. The traditional approach is to completely sever the hardware and software layers so both can be tested individually. Hardware input is collected through servos, and set through an independent interface. Sofware input can be set/read by independently measuring and comparing the signaling that would otherwise go to the hardware and plotting it against known 'good' data.

The tests themselves don't necessarily need to be complex as long as the results are predictable, measurable, and reproducible.

Answer 4

As previously said, mock and stub out the hardware parts. As an example, if you have an interface to the robot, you can inherit from that interface, and then make simple stub implementations of it. Then you can have test that the stub implementation have been called as expected. If that is expected functions or expected parameters.