Problem:
If running all test cases takes too long, I manually specify a subset of tests to run while developing. But when I finally push to CI, it can happen that I broke some other tests or a functionality that is checked by them. So I underselected tests to run. Similarly, sometimes after making changes I don't know which particular tests I should run, so I run a too broad selection (or optimistically leave it to CI :)).
Proposed solution:
A flag in a test runner (for example pytest
) that includes in a test run any test cases which may be affected by what's changed in the project. Not a bulletproof solution, but an approximation of it. It may for example: after every test run store coverage of every test case separately. Then check if lines from each coverage are currently different (or there's a new test) - if so, add it to the testcases to be ran. It may also monitor what non-source files are being accessed and hashes of them.
As you can see there are still cases that some tests may break and they won't be included. But the point is to radically minimize probability that a programmer underselects test cases or waste some time waiting for too broad test to run. If it worked acceptably well, we could by default use only this flag and not define test scope manually at all. Only on CI there would be a full test run.
Question:
Are there any big problems that bury this idea? I know that tests should be fast and the more they touch, the fewer of the kind should be. But saying that we don't want such things "because we should fix the source problem" (the duration of full test suite) is like saying that we don't need tests because we should not make bugs in the first place - not very helpful.
Possibly big problems I see:
- creating a separate coverage for every test case may be a big overhead
- ^ it may not even be possible with current tools
- sum of time costs of such automatically added tests may defeat the purpose (adding something on a module level (in Python) may run a lot of tests)