Where I work, we've been gradually beefing up our automated testing capabilities and coverage.
We're at the point where we've got a wide range of tests, and one configuration of "what needs to be run" is no longer sufficient. We want one set of tests to run when new code is checked in; a larger set for nightly builds; a set that runs before a release; a set that runs with specific hardware attached...
All of our tests are in the form of executables (or, e.g., python scripts that call executables).
- Let's assume, for simplicity's sake, that each executable returns 0 on success, and nonzero on failure.
- Some tests might have additional output (e.g. a unit-test executable will have multiple tests, and be able to export its results to an output file; a higher-scope test might have logs I want to save; etc.).
- The executables are not all from a single type or framework.
- Some test executables might require arguments to define how the test should run; I might define multiple tests, as calls to the same executable with different parameters.
What I'm looking for when I say "test management" is a tool that can do things like:
- Make it easy to add a new test command to a test configuration.
- List what tests are included in a given configuration.
- Nice to have: make it easy to move or duplicate a test command from one configuration to another.
- Run tests, including graceful handling of tests that crash or time out.
- Generate a pass/fail report from a given test run.
- Nice to have: Maintain pass/fail history for individual test commands.
I know tools to manage our tests must exist. But I've had a hard time finding the right tools -- and even more than that, I've had a hard time finding the right terminology. This might just be me, but most resources I've read on testing seem to gloss over the "get the infrastructure up and running" bit of it. So: I'm hoping there's solid terminology I'm missing, or resources on getting infrastructure set up.
TL;DR: What I'm looking for is guidelines and terminology for this part of testing infrastructure. I've got a bunch of test executables, I've got a CI system (or we could switch to a new one, if need be); what are the steps I need to take (and what do I need to read up on) in order for the right sets of tests to run at the right time?