Is there a software engineering principle that relates reuse and regression test cost on a production system?

Question

I've worked on a large financial transaction system for a bank that looked after Pensions and Investments. After 15 years of feature changes, the manual regression test cost had climbed to $200K per release. (10M LOC, $10M transacted per day). This system also interfaces with 19 other systems around the company moving lots of data around. This system was implemented in Java.

What we observe however, is that the more 'reuse' we do, the more regression test costs go up. (The reason being that you need to "test the code you touch" - and reused/shared code impacts a multiplicity of places when it is touched. So despite 'DRY - Do Not Repeat Yourself' - ie don't copy and paste code - we observe a financial incentive to copy and paste code. This is to drive regression test costs down, because we don't want to modify code that could be shared, because that will cause a big regression test impact. )

My question is is there a software engineering principle that describes the relationship between reuse and regression test costs?

The reason I'd ask this question is that arguably there is a cost benefit in decomposing the system into smaller parts to be tested.

Assumptions:

1. 'Regression test' means 'acceptance test' - ie another group spending time to write new and reuse old tests against the system on behalf of the business, including environment and data setups.

2. I know the knee-jerk reaction to a big regression test cost is 'more automated tests'. This is a good principle. In this environment there are a couple of challenges.

   (a) Automated tests are less useful across system boundaries, unless that system has a high automated test coverage as well. (Sphere of influence challenge).

   (b) It is culturally difficult to get momentum on programmer time or capital investment on high automated test coverage when your system is already large and complex.

   (c) The cost of maintaining automated tests is hidden on a project, and so they are easily discarded at a project level.

   (d) This is just the cultural reality of working in a bank.

   (e) I'm working to solve this problem in a different way (decomposition).

Answer 1

we don't want to modify code that could be shared, because that will cause a big regression test impact

Above sounds about right to me. The more important is the code, the more it is shared, the higher are quality requirements, the more quality assurance should be involved when it changes.

Since your system is implemented in Java, you can see an example for above right there, in Java standard libraries (JDK). Its major releases are infrequent, and are accompanied with very effort consuming testing. And even minor releases run through very comprehensive JCK test suite to verify absence of regressions.

You may think this somehow stifles evolution of shared code, and... yes this is about right. The more impact and risk is associated with code change, the more careful you should be about doing it, the more effort needs to be involved into testing its releases.

Ideally, quality of releases of widely shared code should be such that it doesn't need big changes at all (save for infrequent enhancements). This line of thought is reflected in a famous quote of Joshua Bloch:

Public APIs, like diamonds, are forever. You have one chance to get it right so give it your best.

---

With above said though, it looks like some of the problems you describe are caused by inefficient strategy of shared code development. In particular, it looks especially troublesome that for reused code, only two options are considered: either duplicate this code or include it immediately into "core" shared libraries.

Limiting to only these two options is unnecessary and, again, you can find examples of how it can be done better in the very JDK you use. Take a look at java.util.concurrent packages (JSR 166) - until Java 5 release, these were a separate library, not a part of core JDK releases.

Think of it, that's a third option you overlooked, and fairly pragmatic one, the one you need to consider at "inception" of new shared code. When you just figure some code that can be shared between 2-3 components, nothing forces you to immediately include it into core API of the system.

You can package and release this "immature" shared code as separate library, just as it was done to Java concurrent utilities. This way relieves you from the need of full regression testing, because you can use only a relatively small amount of involved components. As a result, you have more leeway to modify and improve this shared code and to test how it works in production.

After your library matures and stabilizes enough to give you confidence that further changes in it are unlikely, you can consider its inclusion into core libraries of the system, just like concurrent utilities have been eventually included into JDK.

---

A concrete example of how much effort (including testing) may be involved into changing heavily reused code can be found, again, in JDK. In release 7u6 they changed internal String representation that involved a change in substring performance. Comments of a feature developer at Reddit outline how much effort was involved in this change:

The initial analysis came out of the GC group in 2007...

Internally the Oracle performance team maintains a set of representative and important apps and benchmarks which they use to evaluate performance changes. This set of apps was crucial in evaluating the change to substring. We looked closely at both changes in performance and change in footprint. Inevitably, as is the case with any significant change, there were regressions in some apps as well as gains in others. We investigated the regressions to see if performance was still acceptable and correctness was maintained...

My reply is not intended to be exhaustive but is a very brief summary of what was almost six months of dedicated work...

Answer 2

I don't think there exist any metrics to calculate "cost for regression tests / LOC of reused code built". And I don't think anyone had ever invested so much time and money to build the same "big" system twice, one version with a lot of resuable components, and one without, to make any serious research on that.

But I have seen problems caused by reuse like yours before, and maybe you are interested in some thoughts about how to handle this better.

First, it is actually not reuse which is your problem - it is rather the attempt to build your own reusable components and use them throughout your system. I am sure you are doing a lot of reuse of big software packages where your problems don't arise: think of the whole Java stack you are using, or maybe some third party components (assumed you were satisfied with that components). But what is different with that software, for example, the Java libraries, while your own reusable components are causing you so much additional regression testing costs? Here are some points I guess that could be different:

• those components are very mature and stable

• they are developed and fully tested in isolation, by a completely different organization

• to (re)use them, you don't have to change them (in fact you could not even if you would like that, since you don't maintain the source code)

• you don't get daily a new version, only minor updates (at maximum per month), or major updates in intervals per year

• most updates are designed to be 100% downwards compatible, especially minor updates

So to make your own reuseable components more successful, you should adapt some of that things from above for your own development:

• for any reusable component, have a clear responsibility who does the maintenance and make sure that all people who reuse a component can be sure to get a bugfix immediately if problems arise.

• establish a strict versioning and release policies. When evolving a reusable component, don't release it "to everyone" every day (at least, not if that would imply mean to run a full $200K regression test on the system). Instead, let new versions only be published from time to time, and provide mechanisms to let user of that component defer the change to the new version.

• the more often a component is reused, the more important is it that it provides a stable interface and downwards compatible behaviour.

• reusable components need very complete test suites to test them in isolation.

Lots of these things will mean that the cost of building the component itself will increase, but it will also decrease the costs of changes caused by failed regressions.

Answer 3

While there may be an "observable" rise in cost due to more testing being needed, that type of refactoring usually makes the code more maintainable in the future as you are decreasing the technical debt in the system.

This should hopefully reduce future bugs, and make new features or alterations to existing features easier to implement.

By easier, I mean they should take less time and therefore cost less.

Reduce, easier and less are all rather nebulous terms here and any future saving (or rather hoped for saving) is impossible to calculate as it hasn't happened yet.

A simpler code base should allow new employees, or existing employees moving onto the project get up to speed quicker, especially for large systems.

It may also reduce staff turnover in that existing project members morale could be improved.

It's of course not guaranteed that you will get these benefits, but these are things that should be considered alongside costs (like increased testing) that can be measured.

In fact, better code should eventually reduce the testing costs over time, even if there is an initial increase due to what you describe.