Iteratively building a Test Framework

Question

(Upfront, I'm not a test or agile expert, just trying to push us to be better. If I'm wrong on anything I'd love your guy's opinions).

So I've been pushing for us to become a real agile shop and less of a "calling ad-hoc agile" shop. So I'm iteratively trying to improve our process. I've found that its been good to slowly build towards "doing it right": I can get buy in for small changes, or large changes on their own. So we've got slow iterations (about a month) that I've been using to slowly fold more and more in. My personal goal is to get us to a process that support continuous integration: at that point I think we can do things like speed up our iterations at much lower cost.

The next bit that I've sold management on that we need to improve is our testing. We've never had a real solid test strategy, so I'd like to start to build up our capabilities there. We've already introduced "Test Driven Development", using Google Test (unit test) framework, which has brought us very far. But these unit tests are not enough to test the system as a whole. We're supplementing this with our old way of testing, ad-hoc system and functional tests that are really specific to the project, that are rewritten every time, and that are not automated. To make matters worse, our functional and system tests are often more similar to characterization tasks than pass-fail tasks: our system/functional test tools need to be able to support advanced plotting and number crunching, which mean a lot of them are in Matlab shudder.

From what I can tell, a good automated test system has two parts: the Test Framework (something akin to the roboframework, ect) that figures out what tests to run, calls those tests, orgamizes the results, ect...the thing that actually does the automation. That uses the Test Tool(s), the fixture that actually performs your test. I may be viewing this wrong, but the Test Framework is like a huge batch file, and the Test Tools are the tests.

So I see two possible ways to go with "what to do next".

1. Implement (could be writing our own, though I'd prefer to acquire/modify) a Test Framework. Leave our ad-hoc system tests, but let the framework take over automation. The goal would to have a way to hit "Test" and have all the tests run.

2. Figure out a way that we can use to make Test Tools more efficiently and effectively.

I'm not sure what the best way to go is here, or if any of the following matter:

• We do both windows and linux development, though usually not at the same time
• While I want to focus on C++, C with CMake, we also do C# and Matlab development. The ideal framework would support all of these, even if it is different tools.
• The "testers", "test automators", and "developers" are all going to be pulled from the same group of highly-competent developers (for now.)
• I need to support command-line executable and libraries now, but would like to support GUIs (QT and .NET) in the future.

The scope of 2 seems HUGE, though doing 1 first seems like putting the cart before the horse. Do I have a semi-accurate picture of whats going on? I've read a few things that use Google Test as their test framework, but doesn't better fit the definition of test tool? Am I biting off more than I can chew?

Answer 1

In my experience, the biggest hurdle to writing tests is in answering the question "how do I write a test?". If it is difficult or unclear, people won't do it. So, if you make it clear, and make it not-difficult, people will start doing it.

So, how do you do that? You pick an existing framework, install it such that anyone can use it, and start using it. Don't get bogged down in picking the best framework, and certainly don't go down the path of building your own since you don't really know what you want.

You are correct that there are two parts to the system -- the testing framework itself, and a scheduler for running the tests. What you want is to get to the point where someone can write a test, check it in, and it automatically builds (if necessary) and runs.

No matter what platform you're on, and no matter what language you're writing your code in, you can probably use the combination of jenkins plus robotframework to test your software. Are they perfect? Definitely not. But they are easy to install, flexible, actively developed, and most likely can let you start writing tests in only a day. Plus, they work very well together since there's a robotframework plugin for jenkins.

Once you have jenkins and robotframework installed, and you start writing a few tests using those tools, you'll begin to get a more clear picture of what you actually need. You'll see places where these tools solve existing problems today, and you'll find deficiencies that you may want to rectify.

The important thing is, start now with something concrete. Get the ball rolling. Learn from the experience, and then you'll have more information to improve your testing solution in the future.