How to efficiently troubleshoot or test new code when hardware setup to reproduce bugs is difficult or impossible to obtain?

Question

I work at a mid-sized company (150ish employees, ~10 size engineering team), and most of my projects involve interfacing with lab equipment (oscilloscopes, optical spectrum analyzers, etc) for the purpose of semi-automated test applications. I have run into a few different scenarios where I am unable to efficiently troubleshoot or test new code because I no longer or never had the hardware setup available to me.

Example 1: A setup where 10-20 "burn-in" processes are run independently using a bench top type sensor - I was able to obtain one such sensor for testing and could occasionally steal a second for simulating all of the facets of interfacing to multiple devices (searching, connecting, streaming, etc).

Eventually a bug showed up (and ultimately ended up being in the device firmware & drivers) that was very difficult to reproduce accurately with only one unit, but hit near "show stopper" levels when 10-20 of these devices were in use simultaneously. This is still unsolved and is ongoing.

Example 2: A test requiring an expensive optical spectrum analyzer as its core component. The device is pretty old, legacy according to the manufacturer who was acquired by a larger company and basically dissolved, and its only documentation was a long winded (and uninformative) document that seems poorly translated. During initial development I was able to keep the device at my desk, but now its tied up, both physically and in schedule during its 24/7 multi-week tests.

When bugs show up related or unrelated to the device, I often need to go through the trouble of testing code external to the application and fitting it in, or writing code blindly and attempting to squeeze in some testing time in between runs, as much of the program logic requires the OSA and the rest of the test hardware to be in place.

I guess my question is how should I approach this? I could potentially spend time developing device simulators, but figuring that into the development estimate will balloon it more than most would probably appreciate. It may not accurately reproduce all issues either, and it's pretty rare to see the same equipment used twice around here. I could get better at unit testing...etc...I could also be loud about the issue and make others understand that temporary delays will be required, not much more than a headache for Research and Development but usually a perceived as a joke when pitched to manufacturing.

Answer 1

Management understands it will take longer to develop and maintain software when you don't have full access to test hardware. You need to take this into account when doing your estimates. Part of the acceptance criteria for putting your software into production should be that you have a way to maintain the software under most circumstances without stopping manufacturing. If you're practicing TDD, this should happen pretty much naturally.

I used to write software for $60 million aircraft. Obviously, there's a high degree of reliability required, and they are reluctant to give every developer one for their desk. We basically had 5 levels of test environments, with more of the real hardware at each level, up to a full aircraft. I estimate 95% of our software could be developed and debugged only with emulators and unit tests. 95% of the remaining features could be worked on the next level up, and so on.

Try to set up similar levels of test environments for yourself. You can't expect to never need access to the real hardware, but if you've set it up so you can't work on your software's GUI without the hardware available, you're wasting valuable time on an expensive resource (not to mention you have some coupling issues with your architecture). Consider that other developers likely have the same issues as you. I would ask the hardware vendor if they already have emulators or other test resources available.

You also need to change your mindset somewhat if you only have limited access to hardware. Rather than trying to debug your application in the normal serial manner, you often need to write code specifically for the purpose of gathering information as quickly as possible.

For example, perhaps you have a bug and you can think of 10 possible causes. If the only time you can get on a machine is the 15 minutes while the operator is on break, write a Short, Self Contained, Correct (Compilable), Example that triggers the bug and write 10 automated tests using that SSCCE to test your theories and log a bunch of data. Afterward back at your desk you can take as long as you need to sift through the data for your next attempt. The idea is to maximize the utility of your limited time with the hardware.

Answer 2

You're trying to solve a problem that isn't yours to solve.

Management needs to prioritize access to the equipment. That may mean you end up with greater access, but it may also mean you end up with less.

Present the challenges you have in an objective format to your management team and ask them for guidance. Your presentation would be a lot stronger if you collaborated with the others who also need access, so all of you can present your case at the same time.

From there, the company (management) has to prioritize who gets access and when. It's a business decision that they need to make since the (lack of) availability of the resources is impacting business development.

Answer 3

You are effectively coding blind.

If management will not pay for test devices, then there is a high likelihood of bugs, or even the development taking longer than it would if you did have real devices to use.

The cost of the devices needn't be completely allocated to the "development" cycle. Maybe they could be rotated into production use, or as a backup. Could they even be resold second hand elsewhere?

Try and cost the bug fixing phases, in both time and money and show the overall cost to your team/company.

Answer 4

Arguing with your bosses is much easier when you have some numbers, or at least some pros and cons at hand, so my suggestion is trying to make a cost vs. benfit analysis. The rough idea goes like this:

• how much development effort do you expect for writing a device simulator? (Note that a device simulator cannot replace the original hardware 100%, especially when the hardware has some unexpected quirks).

• how much testing / debugging effort do you expect without such a tool? Include the costs for your lab workers because you have to block the hardware for testing purposes. Include also the costs for the time the system cannot be used because of bugs and you have trouble to find the root cause.

• how much will additional hardware for testing cost?

• how much time do you expect you will need blocking the hardware for testing purposes?

Of course, reality might not be so simple, and there are a lot of unknown variable in this equation, but try to make some estimations and where you are unsure, ask other people from your environment.

Present the results to your management, discuss the alternatives and then let them decide.