5

Let's say we have a code base covered with big enough amount of unit tests. We make small change to the code and want to check if tests are still passing. Wouldn't it be great to be able to rerun just tests affected by the change? It seems like every bit of data required to build the list of affected tests is in place: set of changed lines could be taken from VCS, and mapping from source code line to corresponding tests might be determined from coverage statistics.

Does any of an existing unit testing frameworks support such technique?

Update. The point is to get a rapid feedback. CI could and should still run full suite, obviously.

Improve this question
4
  • 1
    Is there any reason you tagged this question with unit-testing ? Unit tests are designed to run extremely fast, it shouldn't take more than a couple seconds to run thousands of them. Did you mean to ask about other types of automated tests?
    – Vincent Savard
    Jan 30, 2019 at 13:51
  • You could do this with truly pure functions / methods.
    – Petras Purlys
    Jan 30, 2019 at 20:37
  • @VincentSavard: a tag like "unit-testing" does IMHO not say "hey, this question is about unit-testing (and the by-the-book interpreation of that word) exclusively". But to make you feel better, I took the freedom and added the tag "test-automation".
    – Doc Brown
    Jan 31, 2019 at 7:22
  • @DocBrown Understood, but the title and the question body as well mentioned unit testing, which is why I asked for clarification.
    – Vincent Savard
    Jan 31, 2019 at 13:34

7 Answers 7

Reset to default
7

No, it wouldn't be a good idea.

The point of a test suite is to ensure that development doesn't introduce defects anywhere without having to reason about which parts of the program affect which. If your unit tests are so slow that you'd like to avoid running them all, then inevitably you will sooner or later stop running them, and lose the benefits that they bring. At that point, they stop being an asset and become a pure liability: they cost effort and bring no value to the table. Don't go down that road.

Improve this answer
2
  • 2
    This answer seems to ignore there are projects large enough in the world where even for "fast unit tests" the mentioned idea could become beneficial. And who says this approach must be literally restricted to unit tests? If my whole test suite runs 4 hours, having a smart way to choose a small subset from it and run it locally before I push a small change back into the VCS (so the full suite will be run at night on the server) is IMHO not a bad idea.
    – Doc Brown
    Jan 31, 2019 at 5:21
  • 1
    Your answer seems to infer that there isn't a tool that can reason about which tests are stale and that as a result of having the said tool, that all the tests would no longer be run. I don't think either of these are true. I agree with @DocBrown - there is definitely mileage in this idea.
    – Robbie Dee
    Jan 31, 2019 at 9:17
4

Yes, this is a know technique that is valuable for getting more relevant test results more quickly. The mild drawback that only selecting possibly-affected tests makes the test result more fragile can be avoided by merely using this information to determine the order of tests. That makes it likely that test failures will be reported quickly, while still having to run the full test suite before the tests are marked as passing.

However, this requires that you have per-test coverage data, and a test runner which can use this coverage data. Toolchains that can pull this off are quite rare, especially in the open source world. As in: I know of no such tools that actually implement this. Depending on the granularity you use for the coverage (e.g. per test versus per test suite, or per line versus per function coverage) this can also require a significant amount of storage.

Finally, there is the practical problem that many changes to the source code will not allow you to tie this change to the correct tests. For example, changing the control flow to execute additional code will not find the tests for this additional code. Worse, changing code that is not executed at run time and therefore has no coverage (like type declarations) has far reaching effects but will not select any tests.

In practice, what most test suites do is to organize the tests by the code they cover. Tests for class Foo are in FooTest. This association makes it unnecessary to have coverage data. Tests might also have tags to allow certain kinds of tests to be included/excluded. That might allow a user to manually select a suitable test subset such as test (FooTest or BarTest) and not #slow. That is what I do to deal with slow-ish test suites. With appropriate care, a selection could also be made by a script.

Improve this answer
3

Yes. For JavaScript there is https://wallabyjs.com/

From their homepage:

The tool is insanely fast, because it only executes tests affected by your code changes and runs your tests in parallel.

Improve this answer
2

The closest thing I've seen to what you describe is a ReSharper feature within the unit test runner:

enter image description here

The green ticks with a question mark identify tests which passed in a previous session but which are now deemed to be stale.

While this is a nice feature, I've never quite trusted it enough to rely on. I much prefer to run all the tests in the general area I've been working.

Some developers like to run all the tests any time a change is made locally but if you're familiar with the code base, this isn't always necessary IMHO, unless you're about to commit to the main branch (which ideally, should itself trigger a complete unit test run).

Improve this answer
0

Most unit testing frameworks have abilities to run a selective set of tests. For example, in Javascript using Mocha/Chai one can do a "describe.only".

This is useful (especially if you have a large number of tests) when adding new code and tests as I may only want to run those specific tests in the short term. A large application with several 1000 unit tests would take a bit of time. To avoid that just narrow the scope and then the tests run instantly.

Short term means while I am focusing on the particular task at hand. The linter should/will flag this as an issue. When I am done, I simply remove the "only" part and run a full set of tests.

Always run the full set of tests prior to check in as your change may have impacted other parts of the code you are not aware of. But its OK to limit the scope of the testing prior so you don't to wait for the 1000s of test to run just to see if you new test passes or fails.

Improve this answer
0

What you are thinking of is running a test suite, which is a developer-defined set of tests that are built around specific functionality. Strictly in terms of unit tests, you could accomplish this by having test suites focused at the class level, and if everything is decoupled and mocked out, you could test just the methods changed. However, this is almost never a good idea.

Your intent seems to be to save time, but running 10 tests vs 100 tests doesn't necessarily take 10x as long, and unless you have a very process heavy test (which suggests complexity that can be broken down) it shouldn't take that much time to run all the tests versus the ones you want. In fact, if your tests do take long, it's probably better to run all your tests anyway since the complexity that is probably there could use some testing.

You will almost always save more time by running all your tests versus fixing something you missed because you didn't run all your tests. In fact, we added tests to our build process for this very reason.

Visual Studio does have the option to turn on live unit tests, which tends to intelligently (for VS) detect what tests have changed and rerun those tests. That may be close to what you want, but I cannot stress enough how you should still run all your tests before committing code.

Improve this answer
0

For .NET there is NCrunch (with its Visual Studio extension) and since VS 2017 Enterprise Edition we have the Live Unit Testing feature.

For Java there is at least Infinitest (with its Eclipse extension).

Improve this answer

Not the answer you're looking for? Browse other questions tagged or ask your own question.