1

Problem:

If running all test cases takes too long, I manually specify a subset of tests to run while developing. But when I finally push to CI, it can happen that I broke some other tests or a functionality that is checked by them. So I underselected tests to run. Similarly, sometimes after making changes I don't know which particular tests I should run, so I run a too broad selection (or optimistically leave it to CI :)).

Proposed solution:

A flag in a test runner (for example pytest) that includes in a test run any test cases which may be affected by what's changed in the project. Not a bulletproof solution, but an approximation of it. It may for example: after every test run store coverage of every test case separately. Then check if lines from each coverage are currently different (or there's a new test) - if so, add it to the testcases to be ran. It may also monitor what non-source files are being accessed and hashes of them.

As you can see there are still cases that some tests may break and they won't be included. But the point is to radically minimize probability that a programmer underselects test cases or waste some time waiting for too broad test to run. If it worked acceptably well, we could by default use only this flag and not define test scope manually at all. Only on CI there would be a full test run.

Question:

Are there any big problems that bury this idea? I know that tests should be fast and the more they touch, the fewer of the kind should be. But saying that we don't want such things "because we should fix the source problem" (the duration of full test suite) is like saying that we don't need tests because we should not make bugs in the first place - not very helpful.

Possibly big problems I see:

  1. creating a separate coverage for every test case may be a big overhead
  2. ^ it may not even be possible with current tools
  3. sum of time costs of such automatically added tests may defeat the purpose (adding something on a module level (in Python) may run a lot of tests)
1
  • 1
    If you are developing on a separate branch then what's the problem to always run every single test in CI? Run locally a categorized subset as often as you can and when you get to a point then commit and keep working for the next point while CI runs everything. If often any change may break anything else then you don't have a problem with tests but a serious design issue. Jun 10, 2018 at 17:01

2 Answers 2

1

You may be interested in incremental test runner like pytest-incremental. pytest-incremental analyses the import statements in your code and test code to derive a dependency graph, which it uses to reorder and deselect tests that are unlikely to fail given the changes made since the last test run.

Note, I've never used this particular pytest plugin, but I've used a similar incremental test runner with nosetests. In my experience, incremental test runner in python generally work well as long as you don't do dynamic imports (i.e. using import or importlib) and you have good test hygiene (i.e. keep your tests and their setups independent).

I usually combine incremental test runner with a directory watcher, so tests runs automatically whenever I save the file in my text editor. A plugin like pytest-watch could be useful here.

Additionally, I'd suggest that you should use pytest mark to mark test cases with different markers. You can then use pytest -m flag to select different markers. For example, you may want to mark all slow/integration tests, so you can do pytest -m "not integration" to skip slow integration tests.

I wouldn't recommend a coverage report from incremental test runs. I'd suggest creating a git hook (or whatever similar functionality in your VCS) or to generate your coverage report in your CI. Coverage reporting usually significantly slows down test anyway, so it's generally undesirable to always enable it during development.

0

I would propose to use the make system for this. Have the test case depend only on those files that are necessary. Then create a file .test_a_successful with the touch command: touch .test_a_successful.

So, essentially you would have the following Makefile:

.test_a_successful: testa.py file1.py file2.py file3.py
        python testa.py
        touch .test_a_successful

Whenever you change file4.py, the test won't be re-run needlessly.

The drawback obviously is that because Python is an interpreted language, you may in the future load file4.py and then forget to include it as a dependency in the Makefile.

If you used a compiled language such as C instead, most have a mechanism of autogenerating the header file dependencies in the Makefile. However, I'm not proposing to use C instead of Python without knowing the full details of your project. Just an interesting thing to consider.

4
  • Previously I thought about creating an explicite hierarchy of tests, so if you run a test, every more general will be ran too (bonus: if there are fails, the lowest in hierarchy will be displayed first, because this is the closest to the problem). But the idea here is to decrease manual work to minimum. With big projects the lists that you proposed will contain tens of files.
    – Ctrl-C
    May 11, 2018 at 15:48
  • Some kind of replacement for makedepend, working for Python might help you. Unfortunately my Googling skills didn't result in anything useful. It may be the case such a program for Python doesn't even exist!
    – juhist
    May 11, 2018 at 15:51
  • How about running the subset of tests you believe are relevant while making the changes but run the whole suite once before pushing? I tend to do that even if my tests are fast, because 1 second is still less than 5.
    – axl
    May 11, 2018 at 16:04
  • @axl Merge the two cases from my first paragraph into one case then :)
    – Ctrl-C
    May 11, 2018 at 16:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.