2

Suppose you have a reasonably large codebase (0.5 - 1 msloc) with a large test-suite (6-7hr single-threaded runtime; with a mix of unit-tests and integration-tests built with different tools). You have a proposed patch (or diff or pull-request) for the codebase, and you want to automatically run the most relevant tests. Running all tests would be too expensive. Running smoke-tests would be too shallow. The goal is to find something in-between. By automating the selection of relevant tests, you can help new developers participate in TDD -- and provide timely pre-merge feedback (as part of code-review).

What techniques or arrangements would you use to identify "relevant"? (A basic example: if a patch modifies "src/(.*).php", then run "phpunit test/{$1}Test.php".) What, if any, existing tools come to mind? Or what new tools are needed? Or is it impossible for tooling to help?

(For context: In my particular use-case, I work with LAMP-based web-apps, so tools or techniques that work with PHP/JS/bash are most helpful. But similar issues probably arise with any large app/deep stack, so examples from other stacks could provide insight.)

Improve this question
1
  • 2
    I'll readily concede that the numbers in the example indicate a need for better design/optimization in the test-suite or the application. But the selection/prioritization of tests is an orthogonal issue; can't addressing that bring benefit to optimized or unoptimized suites?
    – user2114266
    Jul 18, 2014 at 1:37

6 Answers 6

Reset to default
10

There are certain static analysis tools that can help determine "test impact", which can then run the effected tests.

But I can't help but feel that you're solving the symptom, not the problem. When I worked in QA, there was one overriding mantra that has helped me as a developer: "don't trust the developer". Even if I could determine "relevant", I wouldn't trust myself to be correct. I mean I can determine that my code is done and correct, why do I need this whole testing exercise?

If your tests are too expensive, then make them cheaper. It's been a decade since I've used a single-threaded unit test framework. 500kloc should take (generously) 5-10 minutes to run through unit tests. Run the unit tests, catch most of the issues and provide you enough certainty to check the code in.

Integration tests might take longer depending on your domain. They shouldn't take so long that you can't run them overnight. Having a day lag time between check-in and integration tests catching your bugs is good enough for most environments, though I would still encourage you to make the integration tests quicker - either by isolating them to particular integration points, eliminating ones that just duplicate unit tests, or by distributing them to a test farm.

Quality is expensive. Skimp on quality and you get what you pay for.

Improve this answer
2
  • Tackling the underlying test-design and test-performance was the first impulse; some of those improvements helped; some need a rethink. FWIW, unit-tests are weak but not the big problem (~20 min with 2 threads); integration-tests with compulsive wait()s are the killer. In any event, the reason I raise the question: "relevance" or "probability of regression" seems like such a simple, alternative approach -- I can't help wondering why it hasn't been explored more.
    – user2114266
    Jul 18, 2014 at 1:11
  • Thanks for the phrase "test impact"; Googling turns up some interesting items.
    – user2114266
    Jul 18, 2014 at 1:16
2

If your unit tests are taking 6-7 hours to run, something is wrong. They should take a few minutes at most. Note that I say should - I know how difficult this can be in reality. Maybe it's time you start mocking out your objects so that you're not dependent on the filesystem or DB or whatever is slowing you down.

You don't want to have to deal with working out which tests are relevant or which ones should be run all the time - you'll end up spending your time doing that instead of actually doing important stuff, like getting things done.

Improve this answer
1
  • 1
    He explicitly mentioned this wasn't just a unit test but a more comprehensive test suite. Mocking out objects as you describe also hides a significant source of bugs (is the DB/FS returning what you think it is?), and it's a lot of work - probably not a good idea.
    – Eamon Nerbonne
    Jul 22, 2014 at 7:17
2

Regression test selection is the term you are looking for. I have recently started thinking about this myself. Applying the principles of incremental compilation, you can determine with certainty which tests need to be run.

By essentially using algorithms already created for code coverage, you can create a list of dependant files for each given test in your suite. Then, by using version control (or a list of content hashes for each file), you can detect which files have changed, then look up which test to run in the dependancy graph you have created.

This obviously has issues, like if your tests span across multiple different services, which make code coverage harder.

I just posted a question on reddit on this topic. View it here

Improve this answer
1

TL;DR: Create a Dependency DAG from affected modules

Identifying the tests that are impacted by a particular change is the same as identifying when to recompile/relink a object file. Create a dependency directed acyclic graph (DAG) starting at the modified module. You should be able to traverse all imports to identify the what needs to be tested. You can also follow out from there to the relevant unit tests. Integration tests will be somewhat harder. You can either run the integration test if it happens to include the updated module; or you can try doing something more clever like tagging the functions or objects that get called during the tests and use that to determine if it needs to be run or not.

Improve this answer
0

You may want to consider only running tests that have recently failed. Given that the entire set takes 6-7 hours (i.e. is runnable nightly), you could base your tests set on the past few nights' results. If you mix in a few randomly selected tests for broader coverage over the course of the day, you should get a good chance of catching errors.

This idea is simple to implement, leverages the notion that most tests rarely if ever fail (considered a bad sign in testing, but usually that's hard to prevent). You get to keep those low-value tests for your nightly full-coverage test, but run the more valuable tests during the day.

If you want to get really fancy, you could deemphasize tests that are correlated with each other, specifically tests that fail only when some other test fails.

Improve this answer
0

Given that "Relevant" presumably means "Tests which will be affected by this code change"

I don't think that its is possible to determine that set any faster than running all the tests. consider:

runsOnStartupOrStaticConstructor.script
{
    Format("c:");
}
Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.