The Stack Overflow podcast is back! Listen to an interview with our new CEO.
2 Copy edited.
source | link

In my experience, tests do not need total coverage to be helpful. Instead, you start reaping different kinds of benefits as coverage increases:

  • more than 30% coverage (aka a couple of integration tests): if your tests fail, something is extremely broken (or your tests are flaky). Thankfully the tests alerted you quickly! But releases will still require extensive manual testing.
  • more than 90% coverage (aka most of the components have superficial unit tests): if your tests pass, the software is likely mostly fine. The untested parts are edge cases, which is fine for non-critical software. But releases will still require some manual testing.
  • very high coverage of functions/statements/branches/requirements: you're living the TDD/BDDBDD dream, and your tests are a precise reflection of the functionality of your software. You can refactor with high confidence, including large scale architectural changes. If the tests pass, your software is almost release ready,ready; only some manual smoke testing required.

The truth is, if you don't start with BDD you're never going to get there, because the work required to test after coding is just excessive. The issue is not writing the tests, but more so being aware of actual requirements (rather than incidental implementation details) and being able to design the software in a way that is both functional and easy to test. When you write the tests first or together with the code, this is practically free.

Since new features require tests, but tests require design changes, but refactoring also requires tests, you have a bit of a chicken and egg problem. As your software creeps closer to decent coverage, you'll have to do do some careful refactoring in those parts of the code where new features occur, just to make the new features testable. This will slow you down a lot – initially. But by only refactoring and testing those parts where new development is needed, the tests also focus on that area where they are needed most. Stable code can continue without tests: if it were buggy, you'd have to change it anyway.

While you try adapting to TDD, a better metric than total project coverage would be the test coverage in parts that are being changed. This coverage should be very high right from the start, though it is not feasible to test all parts of the code that are impacted by a refactoring. Also, you do reap most of the benefits of high test coverage within the tested components. That's not perfect, but still fairly good.

Note that while unit tests seem to be common, starting with the smallest pieces is not a suitable strategy to get a legacy software under test. You'll want to start with integration tests that exercise a large chunk of the software at once. 

E.g. I've found it useful to extract integration test cases from real-world logfiles. Of course running such tests can take a lot of time, which is why you might want to set up an automated server that runs the tests regularly (e.g. a JenkinsJenkins server triggered by commits). The cost of setting up and maintaining such a server is very small compared to not running tests regularly, provided that any test failures actually get fixed quickly.

In my experience, tests do not need total coverage to be helpful. Instead, you start reaping different kinds of benefits as coverage increases:

  • more than 30% coverage (aka a couple of integration tests): if your tests fail, something is extremely broken (or your tests are flaky). Thankfully the tests alerted you quickly! But releases will still require extensive manual testing.
  • more than 90% coverage (aka most of the components have superficial unit tests): if your tests pass, the software is likely mostly fine. The untested parts are edge cases, which is fine for non-critical software. But releases will still require some manual testing.
  • very high coverage of functions/statements/branches/requirements: you're living the TDD/BDD dream, and your tests are a precise reflection of the functionality of your software. You can refactor with high confidence, including large scale architectural changes. If the tests pass, your software is almost release ready, only some manual smoke testing required.

The truth is, if you don't start with BDD you're never going to get there, because the work required to test after coding is just excessive. The issue is not writing the tests, but more so being aware of actual requirements (rather than incidental implementation details) and being able to design the software in a way that is both functional and easy to test. When you write the tests first or together with the code, this is practically free.

Since new features require tests, but tests require design changes, but refactoring also requires tests, you have a bit of a chicken and egg problem. As your software creeps closer to decent coverage, you'll have to do do some careful refactoring in those parts of the code where new features occur, just to make the new features testable. This will slow you down a lot – initially. But by only refactoring and testing those parts where new development is needed, the tests also focus on that area where they are needed most. Stable code can continue without tests: if it were buggy, you'd have to change it anyway.

While you try adapting to TDD, a better metric than total project coverage would be the test coverage in parts that are being changed. This coverage should be very high right from the start, though it is not feasible to test all parts of the code that are impacted by a refactoring. Also, you do reap most of the benefits of high test coverage within the tested components. That's not perfect, but still fairly good.

Note that while unit tests seem to be common, starting with the smallest pieces is not a suitable strategy to get a legacy software under test. You'll want to start with integration tests that exercise a large chunk of the software at once. E.g. I've found it useful to extract integration test cases from real-world logfiles. Of course running such tests can take a lot of time, which is why you might want to set up an automated server that runs the tests regularly (e.g. a Jenkins server triggered by commits). The cost of setting up and maintaining such a server is very small compared to not running tests regularly, provided that any test failures actually get fixed quickly.

In my experience, tests do not need total coverage to be helpful. Instead, you start reaping different kinds of benefits as coverage increases:

  • more than 30% coverage (aka a couple of integration tests): if your tests fail, something is extremely broken (or your tests are flaky). Thankfully the tests alerted you quickly! But releases will still require extensive manual testing.
  • more than 90% coverage (aka most of the components have superficial unit tests): if your tests pass, the software is likely mostly fine. The untested parts are edge cases, which is fine for non-critical software. But releases will still require some manual testing.
  • very high coverage of functions/statements/branches/requirements: you're living the TDD/BDD dream, and your tests are a precise reflection of the functionality of your software. You can refactor with high confidence, including large scale architectural changes. If the tests pass, your software is almost release ready; only some manual smoke testing required.

The truth is, if you don't start with BDD you're never going to get there, because the work required to test after coding is just excessive. The issue is not writing the tests, but more so being aware of actual requirements (rather than incidental implementation details) and being able to design the software in a way that is both functional and easy to test. When you write the tests first or together with the code, this is practically free.

Since new features require tests, but tests require design changes, but refactoring also requires tests, you have a bit of a chicken and egg problem. As your software creeps closer to decent coverage, you'll have to do some careful refactoring in those parts of the code where new features occur, just to make the new features testable. This will slow you down a lot – initially. But by only refactoring and testing those parts where new development is needed, the tests also focus on that area where they are needed most. Stable code can continue without tests: if it were buggy, you'd have to change it anyway.

While you try adapting to TDD, a better metric than total project coverage would be the test coverage in parts that are being changed. This coverage should be very high right from the start, though it is not feasible to test all parts of the code that are impacted by a refactoring. Also, you do reap most of the benefits of high test coverage within the tested components. That's not perfect, but still fairly good.

Note that while unit tests seem to be common, starting with the smallest pieces is not a suitable strategy to get a legacy software under test. You'll want to start with integration tests that exercise a large chunk of the software at once. 

E.g. I've found it useful to extract integration test cases from real-world logfiles. Of course running such tests can take a lot of time, which is why you might want to set up an automated server that runs the tests regularly (e.g. a Jenkins server triggered by commits). The cost of setting up and maintaining such a server is very small compared to not running tests regularly, provided that any test failures actually get fixed quickly.

1
source | link

In my experience, tests do not need total coverage to be helpful. Instead, you start reaping different kinds of benefits as coverage increases:

  • more than 30% coverage (aka a couple of integration tests): if your tests fail, something is extremely broken (or your tests are flaky). Thankfully the tests alerted you quickly! But releases will still require extensive manual testing.
  • more than 90% coverage (aka most of the components have superficial unit tests): if your tests pass, the software is likely mostly fine. The untested parts are edge cases, which is fine for non-critical software. But releases will still require some manual testing.
  • very high coverage of functions/statements/branches/requirements: you're living the TDD/BDD dream, and your tests are a precise reflection of the functionality of your software. You can refactor with high confidence, including large scale architectural changes. If the tests pass, your software is almost release ready, only some manual smoke testing required.

The truth is, if you don't start with BDD you're never going to get there, because the work required to test after coding is just excessive. The issue is not writing the tests, but more so being aware of actual requirements (rather than incidental implementation details) and being able to design the software in a way that is both functional and easy to test. When you write the tests first or together with the code, this is practically free.

Since new features require tests, but tests require design changes, but refactoring also requires tests, you have a bit of a chicken and egg problem. As your software creeps closer to decent coverage, you'll have to do do some careful refactoring in those parts of the code where new features occur, just to make the new features testable. This will slow you down a lot – initially. But by only refactoring and testing those parts where new development is needed, the tests also focus on that area where they are needed most. Stable code can continue without tests: if it were buggy, you'd have to change it anyway.

While you try adapting to TDD, a better metric than total project coverage would be the test coverage in parts that are being changed. This coverage should be very high right from the start, though it is not feasible to test all parts of the code that are impacted by a refactoring. Also, you do reap most of the benefits of high test coverage within the tested components. That's not perfect, but still fairly good.

Note that while unit tests seem to be common, starting with the smallest pieces is not a suitable strategy to get a legacy software under test. You'll want to start with integration tests that exercise a large chunk of the software at once. E.g. I've found it useful to extract integration test cases from real-world logfiles. Of course running such tests can take a lot of time, which is why you might want to set up an automated server that runs the tests regularly (e.g. a Jenkins server triggered by commits). The cost of setting up and maintaining such a server is very small compared to not running tests regularly, provided that any test failures actually get fixed quickly.