30

I work at a mid-sized company (150ish employees, ~10 size engineering team), and most of my projects involve interfacing with lab equipment (oscilloscopes, optical spectrum analyzers, etc) for the purpose of semi-automated test applications. I have run into a few different scenarios where I am unable to efficiently troubleshoot or test new code because I no longer or never had the hardware setup available to me.

Example 1: A setup where 10-20 "burn-in" processes are run independently using a bench top type sensor - I was able to obtain one such sensor for testing and could occasionally steal a second for simulating all of the facets of interfacing to multiple devices (searching, connecting, streaming, etc).

Eventually a bug showed up (and ultimately ended up being in the device firmware & drivers) that was very difficult to reproduce accurately with only one unit, but hit near "show stopper" levels when 10-20 of these devices were in use simultaneously. This is still unsolved and is ongoing.

Example 2: A test requiring an expensive optical spectrum analyzer as its core component. The device is pretty old, legacy according to the manufacturer who was acquired by a larger company and basically dissolved, and its only documentation was a long winded (and uninformative) document that seems poorly translated. During initial development I was able to keep the device at my desk, but now its tied up, both physically and in schedule during its 24/7 multi-week tests.

When bugs show up related or unrelated to the device, I often need to go through the trouble of testing code external to the application and fitting it in, or writing code blindly and attempting to squeeze in some testing time in between runs, as much of the program logic requires the OSA and the rest of the test hardware to be in place.

I guess my question is how should I approach this? I could potentially spend time developing device simulators, but figuring that into the development estimate will balloon it more than most would probably appreciate. It may not accurately reproduce all issues either, and it's pretty rare to see the same equipment used twice around here. I could get better at unit testing...etc...I could also be loud about the issue and make others understand that temporary delays will be required, not much more than a headache for Research and Development but usually a perceived as a joke when pitched to manufacturing.

2
  • 5
    a device simulator (or mockable interface) will pay for itself just in the convenience May 19, 2014 at 16:00
  • 21
    @ratchetfreak - as one who spends his days simulating devices (I work on a medical device simulator full time), let me assure you that even a low-fidelity simulation of someone else's piece of equipment can be a VERY difficult undertaking, depending on the connections, protocols, and data types involved. If the test equipment the OP uses is anything like the the gear I have to deal with, it can take days or weeks of poking at it to just figure out what the bloody thing is REALLY doing (as opposed to what the spec says). So it's not at ALL a foregone conclusion that a simulator is worth it. May 19, 2014 at 16:36

4 Answers 4

35

Management understands it will take longer to develop and maintain software when you don't have full access to test hardware. You need to take this into account when doing your estimates. Part of the acceptance criteria for putting your software into production should be that you have a way to maintain the software under most circumstances without stopping manufacturing. If you're practicing TDD, this should happen pretty much naturally.

I used to write software for $60 million aircraft. Obviously, there's a high degree of reliability required, and they are reluctant to give every developer one for their desk. We basically had 5 levels of test environments, with more of the real hardware at each level, up to a full aircraft. I estimate 95% of our software could be developed and debugged only with emulators and unit tests. 95% of the remaining features could be worked on the next level up, and so on.

Try to set up similar levels of test environments for yourself. You can't expect to never need access to the real hardware, but if you've set it up so you can't work on your software's GUI without the hardware available, you're wasting valuable time on an expensive resource (not to mention you have some coupling issues with your architecture). Consider that other developers likely have the same issues as you. I would ask the hardware vendor if they already have emulators or other test resources available.

You also need to change your mindset somewhat if you only have limited access to hardware. Rather than trying to debug your application in the normal serial manner, you often need to write code specifically for the purpose of gathering information as quickly as possible.

For example, perhaps you have a bug and you can think of 10 possible causes. If the only time you can get on a machine is the 15 minutes while the operator is on break, write a Short, Self Contained, Correct (Compilable), Example that triggers the bug and write 10 automated tests using that SSCCE to test your theories and log a bunch of data. Afterward back at your desk you can take as long as you need to sift through the data for your next attempt. The idea is to maximize the utility of your limited time with the hardware.

3
  • Accepted this answer as it was the most complete - and I think a good balance of "make management aware" with "change your practices". I think spending some effort on better levels of decoupling and some level of hardware simulators should be worthwhile and I can show this in my estimates. I also especially like the tips for some squeezing in some quick full featured tests that capture a lot of data while debugging - thank you.
    – plast1k
    May 19, 2014 at 19:01
  • 14
    I stopped reading after "Management understands"
    – PlasmaHH
    May 19, 2014 at 21:05
  • 1
    "Reluctant to give every developer one for their desk". Ironically, you could probably bend the numbers enough to prove that giving each developer their own $60 million plane to work on would be cheaper than the total cumulative cost of an airline disaster! May 20, 2014 at 3:06
15

You're trying to solve a problem that isn't yours to solve.

Management needs to prioritize access to the equipment. That may mean you end up with greater access, but it may also mean you end up with less.

Present the challenges you have in an objective format to your management team and ask them for guidance. Your presentation would be a lot stronger if you collaborated with the others who also need access, so all of you can present your case at the same time.

From there, the company (management) has to prioritize who gets access and when. It's a business decision that they need to make since the (lack of) availability of the resources is impacting business development.

1
  • 4
    One thing that might help in talking to management is to predicate your schedules (or your milestones) on equipment access. You can only do so much without the hardware in front of you, and if you make it clear that the estimate is from when they give it to you, then management can make decisions with full knowledge. May 19, 2014 at 16:37
4

You are effectively coding blind.

If management will not pay for test devices, then there is a high likelihood of bugs, or even the development taking longer than it would if you did have real devices to use.

The cost of the devices needn't be completely allocated to the "development" cycle. Maybe they could be rotated into production use, or as a backup. Could they even be resold second hand elsewhere?

Try and cost the bug fixing phases, in both time and money and show the overall cost to your team/company.

4

Arguing with your bosses is much easier when you have some numbers, or at least some pros and cons at hand, so my suggestion is trying to make a cost vs. benfit analysis. The rough idea goes like this:

  • how much development effort do you expect for writing a device simulator? (Note that a device simulator cannot replace the original hardware 100%, especially when the hardware has some unexpected quirks).

  • how much testing / debugging effort do you expect without such a tool? Include the costs for your lab workers because you have to block the hardware for testing purposes. Include also the costs for the time the system cannot be used because of bugs and you have trouble to find the root cause.

  • how much will additional hardware for testing cost?

  • how much time do you expect you will need blocking the hardware for testing purposes?

Of course, reality might not be so simple, and there are a lot of unknown variable in this equation, but try to make some estimations and where you are unsure, ask other people from your environment.

Present the results to your management, discuss the alternatives and then let them decide.

3
  • I think you meant can't in here Note that a device simulator can seldom replace the original hardware 100%, especially when the hardware has some unexpected quirks
    – Rémi
    Mar 16, 2015 at 12:11
  • @Rémi: maybe "can seldom" is not the usual order of words in plain english? FWIW, I changed my answer to make this unambigous, thanks for the reply.
    – Doc Brown
    Mar 16, 2015 at 12:49
  • I do not speak English natively but it read strange. thanks
    – Rémi
    Mar 16, 2015 at 13:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.