Where I work, we've been gradually beefing up our automated testing capabilities and coverage.

We're at the point where we've got a wide range of tests, and one configuration of "what needs to be run" is no longer sufficient. We want one set of tests to run when new code is checked in; a larger set for nightly builds; a set that runs before a release; a set that runs with specific hardware attached...

All of our tests are in the form of executables (or, e.g., python scripts that call executables).

  • Let's assume, for simplicity's sake, that each executable returns 0 on success, and nonzero on failure.
  • Some tests might have additional output (e.g. a unit-test executable will have multiple tests, and be able to export its results to an output file; a higher-scope test might have logs I want to save; etc.).
  • The executables are not all from a single type or framework.
  • Some test executables might require arguments to define how the test should run; I might define multiple tests, as calls to the same executable with different parameters.

What I'm looking for when I say "test management" is a tool that can do things like:

  • Make it easy to add a new test command to a test configuration.
  • List what tests are included in a given configuration.
  • Nice to have: make it easy to move or duplicate a test command from one configuration to another.
  • Run tests, including graceful handling of tests that crash or time out.
  • Generate a pass/fail report from a given test run.
  • Nice to have: Maintain pass/fail history for individual test commands.

I know tools to manage our tests must exist. But I've had a hard time finding the right tools -- and even more than that, I've had a hard time finding the right terminology. This might just be me, but most resources I've read on testing seem to gloss over the "get the infrastructure up and running" bit of it. So: I'm hoping there's solid terminology I'm missing, or resources on getting infrastructure set up.

TL;DR: What I'm looking for is guidelines and terminology for this part of testing infrastructure. I've got a bunch of test executables, I've got a CI system (or we could switch to a new one, if need be); what are the steps I need to take (and what do I need to read up on) in order for the right sets of tests to run at the right time?

  • As said cauchy Jenkins might do the job, for the terminology Jenkins is a Continous Integration (CI) Platform
    – Walfrat
    Commented Jun 28, 2017 at 13:32
  • How do you run your existing tests currently? Some xUnit testing tool? Just some higher-level script running lower-level scripts?
    – Doc Brown
    Commented Jun 28, 2017 at 18:21
  • @DocBrown: Our existing tests (that run automatically) are gtest executables. During CI, we run a script calling them all (and handling timeouts; crashes; etc.). We can tweak the script to allow for a wider variety of tests, but it's at the point where it's getting complex enough for me to look for established tools.
    – Standback
    Commented Jun 29, 2017 at 5:37
  • In the end I wound up using CTest, which does exactly what I was looking for here. However, since the question is one of definition and terminology, not a tool-request (and since CTest is pretty specific to certain environments), I'm leaving the question open.
    – Standback
    Commented May 2, 2019 at 17:48

2 Answers 2


My two cents. You say:

We want one set of tests to run when new code is checked in; a larger set for nightly builds; a set that runs before a release; a set that runs with specific hardware attached

Each of those cases is different. Threat them differently. I would recommend taking a look at jenkins (or a similar tool). You can have a server running jenkins, which could:

  • Launch a script every night.
  • Launch a script when you commit to the repository.

Even though I think the last is possible to do in theory, I was not able to do so when I tried. So an alternative could be to add the script to your repository, and before pushing to master, all team members must run the script themselves. If somebody fails to do so and break the build, you will catch it at midnight. Alternative this script could be run several times a day, so you catch it more quickly.

  • For the other two options (special hardware or release), you could add those scripts to the repository too and run them manually.

If you already have python scripts that do what you say, all you are missing is putting some order in the house.

The JUnit XML format is recognized by jenkins and other tools. If you manage to write the output of your python scripts in this format it will help you a lot in the long run. I'm not sure you can use jenkins if you don't.

Again, don't look for a "one answer to solve them all". Those tests cases are different, look for tools for each case.

  • So, we're currently using a script to run our testing. I want to clarify: We're already reducing each test to a one-line command; each in its own way. But the management of all the tests seems complex enough that I want to look for existing tools. I want it to be easy to add a new test; I'd like an easy way to see what tests are in what configurations; I want a pass/fail summary per run; maybe a pass/fail history per test... this feels like a management system I'd expect to exist.
    – Standback
    Commented Jun 29, 2017 at 5:43

First, your post reads a little bit like a question for a tools recommendation in disguise (but I am sure you know that this would be off-topic for this site). If you google for "test management system", you find plenty of links (like this one: 15 Best Test Management Tools of 2017).

The main part of what you are trying to achieve can be implemented by using (maybe "abusing") a typical unit testing tool like NUnit or JUnit (or the module "unittest" for Python). For example, in my dev team, we are actually using NUnit with its UI - not just for unit tests, but also for any kind of automated test, including ones where arbitrary executables are included. NUnit gives us the following capabilities:

  • you can easily group and categorize tests and run (or exclude) tests by category
  • the UI gives you a overview over all existing tests, and you can pick individual ones or subsets and run them interactively
  • you can run tests either by GUI or command line (so you could integrate this easily into any CI system)
  • you get a pass/fail report after each run

Of course, tests running arbitrary executables, especially with graceful handling in case of a crash or time out, is nothing you get "out of the box", for this part we implemented a little bit of infrastructure by ourselves.

If you decide to go a similar route, adding new tests to a configuration, or moving tests from one configuration to another will become easy over time. Just develop some conventions for your team, write some short documentation and provide some examples.

As you already noted by yourself, it helps to standardize the tests, for example, by making each test a command line program with similar parameters and standard behaviour, by using consistent naming conventions for tests and folders, and by putting output files of individual tests always in a specific folder (for example, in a subfolder below the folder where the executable is, or a temp folder with an according name). But that is nothing a test management tool will do for you, this part you will have to work out for yourself.

  • Yeah, I'm kind of trying to thread a needle here: I feel like I'm missing really basic terminology, and that's hampered my attempts to grok tools on my own. This might just be me, but most resources I've read on testing seem to gloss over the "get the infrastructure up and running" bit of it. For example, a lot of the tools in the "Best Tools" link seem QA oriented, and it isn't at all clear to me if/how they handle unit testing, or what they're assuming about our codebase and setup.
    – Standback
    Commented Jun 29, 2017 at 6:19
  • So: I'm hoping there's solid terminology I'm missing, or resources on getting infrastructure set up, rather than "I need X,Y,Z, please choose a tool for me."
    – Standback
    Commented Jun 29, 2017 at 6:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.