Selecting test cases to run by checking what's changed

Question

Problem:

If running all test cases takes too long, I manually specify a subset of tests to run while developing. But when I finally push to CI, it can happen that I broke some other tests or a functionality that is checked by them. So I underselected tests to run. Similarly, sometimes after making changes I don't know which particular tests I should run, so I run a too broad selection (or optimistically leave it to CI :)).

Proposed solution:

A flag in a test runner (for example pytest) that includes in a test run any test cases which may be affected by what's changed in the project. Not a bulletproof solution, but an approximation of it. It may for example: after every test run store coverage of every test case separately. Then check if lines from each coverage are currently different (or there's a new test) - if so, add it to the testcases to be ran. It may also monitor what non-source files are being accessed and hashes of them.

As you can see there are still cases that some tests may break and they won't be included. But the point is to radically minimize probability that a programmer underselects test cases or waste some time waiting for too broad test to run. If it worked acceptably well, we could by default use only this flag and not define test scope manually at all. Only on CI there would be a full test run.

Question:

Are there any big problems that bury this idea? I know that tests should be fast and the more they touch, the fewer of the kind should be. But saying that we don't want such things "because we should fix the source problem" (the duration of full test suite) is like saying that we don't need tests because we should not make bugs in the first place - not very helpful.

Possibly big problems I see:

1. creating a separate coverage for every test case may be a big overhead
2. ^ it may not even be possible with current tools
3. sum of time costs of such automatically added tests may defeat the purpose (adding something on a module level (in Python) may run a lot of tests)

Answer 1

You may be interested in incremental test runner like pytest-incremental. pytest-incremental analyses the import statements in your code and test code to derive a dependency graph, which it uses to reorder and deselect tests that are unlikely to fail given the changes made since the last test run.

Note, I've never used this particular pytest plugin, but I've used a similar incremental test runner with nosetests. In my experience, incremental test runner in python generally work well as long as you don't do dynamic imports (i.e. using import or importlib) and you have good test hygiene (i.e. keep your tests and their setups independent).

I usually combine incremental test runner with a directory watcher, so tests runs automatically whenever I save the file in my text editor. A plugin like pytest-watch could be useful here.

Additionally, I'd suggest that you should use pytest mark to mark test cases with different markers. You can then use pytest -m flag to select different markers. For example, you may want to mark all slow/integration tests, so you can do pytest -m "not integration" to skip slow integration tests.

I wouldn't recommend a coverage report from incremental test runs. I'd suggest creating a git hook (or whatever similar functionality in your VCS) or to generate your coverage report in your CI. Coverage reporting usually significantly slows down test anyway, so it's generally undesirable to always enable it during development.

Answer 2

I would propose to use the make system for this. Have the test case depend only on those files that are necessary. Then create a file .test_a_successful with the touch command: touch .test_a_successful.

So, essentially you would have the following Makefile:

.test_a_successful: testa.py file1.py file2.py file3.py
        python testa.py
        touch .test_a_successful

Whenever you change file4.py, the test won't be re-run needlessly.

The drawback obviously is that because Python is an interpreted language, you may in the future load file4.py and then forget to include it as a dependency in the Makefile.

If you used a compiled language such as C instead, most have a mechanism of autogenerating the header file dependencies in the Makefile. However, I'm not proposing to use C instead of Python without knowing the full details of your project. Just an interesting thing to consider.