Software Engineering Stack Exchange is a question and answer site for professionals, academics, and students working within the systems development life cycle. It only takes a minute to sign up.
Sign up to join this community
In a blog post by Andrew Hay, the following axiom was posited:
It costs significantly more to fix a bug at the end of the project that it does to fix the same bug earlier in the project.
However, this doesn't seem certain, especially after reading a blog post on Less Wrong, and the data I've seen to back it up is extremely old.
Is this still axiom accurate today?
The only hard data I've ever seen is Boehm and Papaccio, Understanding and Controlling Software Costs.
This stems back to 1988 and was a study of around 80 software projects. They concluded that a decision made early and corrected late could cost 50-200 times what it would have if it had been corrected early. But the kind of very early decisions they're talking about are which OS to run on and which language and database to use.
So those figures may be overwrought with respect to today's software development. However, now we have a lot of experience in the field and we know instinctively that it still holds true to a point.
In the extreme, we know that if a failure in requirements is caught just before we go to production, this causes a lot of rework and get the project delayed or indeed cancelled, where if it had been caught before any work had been done, we'd have been fine.
Edit: Doc Brown makes a good point in his comment.
Boehm's research was done on COBOL and FORTRAN projects in a time when compile and run times were ridiculously slow. I started my career in the early 90s on COBOL and the compile and test cycle used to take so long that it was worth the effort to dry-test the code before going through the cycle (or at least during the build phase, just in case you could catch something and cancel it early, saving yourself an hour or so).
In turn, our bosses used to laugh at our complaints because it wasn't so long ago they used to have to carry a box of hand-sorted punch cards to the server room and leave it there for a day.
So it was definitely more true then than it is now.
And yet, very recently, I've seen blogs reusing Steve McConnell's visualisation of this problem (ref, dated 1996) as if that graph was actually based on hard numbers. It isn't. It's a visualisation, to explain his point simply.
I think that Morendil's premise, in the article the OP quoted, is a good one. The science we have on this subject is poor and outdated and yet treated as canon. But I also think that it holds up well and sounds true because we know from bitter experience that it is still true, at least to a point. And I think his dramatic "diseased discipline" phrasing does him no favours.
While I'm not aware of any hard data or other evidence to support this claim, at a bare minimum, I'd figure it's common sense.
Think about it this way.. If you have a complex system with inter-dependent subsystems (as most non-trivial applications do), think about knock-on issues that may be a result of changes made to any one of the systems. If subsystems are proven to be correct (via unit tests and the like) and fixed early, the number of bugs that will be caused due to knock-ons alone are mitigated simply by fixing early.
Also, if you're fixing bugs early, the implementation is still fresh in the developer's mind. Depending on the length of any given project, if you're fixing bugs at the end, the developer will need to spend time figuring out what they wrote and (perhaps) how the subsystems that their code depends on works. Time spent re-learning this = $.
I doubt that it's at all possible to come up with a scientifically rigid way of measuring this - there's just too many other factors involved, and no two projects are comparable enough to serve as more than case studies. Logical thinking should get you a long way though. A few arguments:
This is basic accepted stuff from the systems engineering - and it applies to any form of technical development (be that building bridges, missiles, battleships or software).
Essentially, the cost of things goes up approximately an order of magnitude as you move through stages of development.
Something that costs $10 to fix at the point of conceiving the idea...
Will cost about $100 if you need to go an update the specification....
Or cost about $1000 if something was implemented and you need to make changes at that point (and update the spec, and get the approvals and so on) but it had not gone through some kind of formal acceptance / sell off test
Or cost about $10000 if something was implemented and customer accepted, and you need to make changes at that point (and update the spec, and get the approvals, and re-test and re-run customer acceptance and qualification, and so on)
And the cost after deployment / roll out / putting in service is even more again.
Examples abound and its easy to understand them: a banking system with a serious scope change made after you have 25,000 employees using it will cost a packet in re-training time... before you even consider the scoping, coding, test, regression, etc etc etc.
OBVIOUSLY your mileage will vary: the costs and impacts of changing Fred Nurke's electronic sock warming e-commerce web site are somewhat different to the costs of changing software on an aircraft flight control computer.
I don't have access to hard data or facts, so I can only offer you anecdotal observations gleaned from my last 20 years in IT.
I believe there is a vast difference between the that way most developers create software today as compared to 20 years ago. With the Agile movement having gained so much momentum, particularly in the last 5-6 years, I've seen a real shift in attitudes in the workplace. So much so that the quality of what we do just seems to grow in leaps and bounds every year, and with every project as we apply the lessons we've learned from project to project. Leaner processes combined with a focus on test-first development has grown from being very controversial to commonplace. So much so that to walk into many companies today, if you are not comfortable with Agile you'll be lucky if they don't show you the door.
So what impact has this had. First of all, I have noticed that problems do often get identified much earlier. Often it is the case that if the problem doesn't look to be too great, it can sometimes be put on hold indefinitely. In a rare handful of cases, I have seen bugs that were thought to be trivial become serious problems when addressed later, as some fundamental issue becomes apparent that wasn't considered at the time. Sometimes this can lead to a drawn out fix cycle, and that can be costly to a degree, but that cost is often measured less in terms of resourcing, and more often in terms of the impact on the relationship between customer and developer. Customers are growing used to this Agile way of thinking, which returns results to them much faster than it did in the old days, with highly iterative development sprints and fast turnaround between requests and implementation, so they have come to expect a great deal of us. And as far as the actual bugs are concerned, the time to get a bug fixed is more often greatly diminished as a result of having a solid suite of tests to support changes, and the ability to create new tests from which to provide insight and solutions to the problems reported.
So on the whole, it appears that the overall effort to fix bugs has been in most cases reduced if there is a robust suite of tests in place, and procedures to ensure that testing remains the focus of what the developer does, but the actual cost has in some ways shifted in part at least from the implementation, to other areas of the business, because in some ways, the focus has also shifted from pure supply and demand to relationship management.
Another thing that has become apparent, is that our gut instincts of a few years ago which suggested that being Agile would reduce our maintenance cycles has been proven to a degree both right and wrong. Right in the sense that solid testing has made it easier to debug and fix our code to a large degree, and to reduce overall the number of bugs released into production code, and wrong in the sense that we are now working harder to avoid needing to maintain legacy code, by constantly refactoring code and improving architecture such that it is becoming rarer that we need to develop new products completely from scratch.
So in the end, what does this mean with regards to the OP's question? Well, it means that the answer really isn't as cut-and-dry as we once might have thought it to be. 15 years ago, I would have probably answered the question as a Yes, but now I feel it's more realistic to say that it really is too difficult to measure empirically, because the nature of what we do to develop software has changed greatly from when we first started asking ourselves the OP's question back then. In some ways, the more we advance our techniques and skills as an industry, the further the question grows from a definitive yes, to a point where I suspect that in a short number of years we'll be saying that it doesn't matter when we fix bugs, because our tests and processes will be so much more robust, that the timing of the bug fixes will be less driven by efforts to save our budgets, and more by priorities to satisfy our customers needs, and the relative cost will become virtually meaningless contextually.
But as I say, this isn't hard data-supported evidence, just my observations of the past few years, and my gut telling me that there will be more ground-shaking wisdom to come that will improve the way that we do things.
Early bugs will propagate to other parts of the system so when you fix the bug you may be forced to rewrite some parts of the system that relied on the bug itself.
Beside over time you'll foget how some parts of the program are built and you'll need to remind yourself. It's some form of technical debt (if you rush the project in early stage you'll have problems with finishing it because of shortcuts you have taken).
It's as simple as that and there's nothing to proof.
I think you're trying to rush the project as fast as possible to present some working solution to your employee. Good news is that you'll have it very fast, bad news is that you probably will never finish it without complete rewrite if you just keep writing crap as fast as possible and plan to fix everything in a couple of months. You probably won't even be able to refactor this.
Well, I probably can't give you the definitive proof you're asking for, but I can relate a fairly recent incident from my work.
We added a feature that provided workflow management capabilities to our product. Typical BDUF stuff, specs signed off and approved by the client. Implemented to spec. Complaints from Day 1 on deployment.
We hadn't done a real usability walkthrough with the client, just taken their word for what they wanted. Result: hundreds of hours of rework - analysis, design, implementation, and QA had to be redone. All because the spec missed particular use cases. A bug in the specification, if you will.
I've seen similar things in prior jobs when someone in the chain makes assumptions that are different from the end-user assumptions. Straight-up coding bugs are relatively easy to deal with if caught close to when they happen, but design bugs can kill entire systems.
If you fix the bug after release, then you have the cost of finding and fixing the bug - which may or may not take more time/cost to do post release. However you have a round of Integration testing, regression testing, UA testing, release activities, etc that will need to be accounted for. Unless the bug fix goes in with a number of other fixes or a version update you will have additional expense for the testing and release activities that would be avoided by including the fix in the initial release - because these costs would be shared over a number of fixes/updates/features.
Also consider the cost that the bug will cause in use, if is just cosmetic, then it probably does not matter, but a function or performance bug could create a cost with support activities or reduced productivity or incorrect calculations.
Ask Intel how much the Pentium Bug cost them, The Ariane 5 Rocket is another good example. These bugs were fixed at the end oft he project. I have worked on a system where an "attempt" at a software release has a budget of 6 figures. In these extreme cases, it's easy to see the cost. In other (most?) cases, the cost gets hidden by the noise, but it is still there.
There is no doubt that bugs cost money while they exist. One item, Defect reports, take time to compile, triage and close as dup, time is money- therefore an open bug creates ongoing cost. therefore, it must be that deferring bug fixes costs more than fixing them sooner.
If a bug escapes into the wild, the cost has a step wise jump...... Is "The end of the project" before or after the release of the software?
I once read an article that had two interesting points (unfortunately the references I had is long gone, so I'll have to just postulate here). The first point they made was that around 50% of all errors were introduced in the requirements specification and that around 90% of all errors where found during UAT or System tests.
The second point they had was that for each phase in the V-model the cost was increased 10-fold. Whether or not the factor is correct I find kind of irrelevant but the most costly errors are when your design is based on an incorrect assumption. This leads to a massive amount of rewritting. All the code that works because of that assumption but fails when the correct assumption is applied will have to be rewritten.
I've experienced the entire domain model having to be rewritten because of one incorrect assumption in the requirements specifications. If such a bug is caught early, Ie when reviewing the requirements specifications the cost is very low. In this particular case it would have taken ten lines of text. In the case where it's found during UAT (as this was) the cost is sumstantial (in the given example the project cost was increased by 50%)
No statistical data, but personal experience:
The rocket motor control code I was working on had a line like powerCutoff = someCondition && debugOptions.cutoffIsAllowed;. The default option was no cutoff allowed. The 'final' build was supposed to remove all the debug options, so the line got modified to powerCutoff = someCondition;.
Did you catch the bug during code review? We didn't. The first time the trigger condition occurred in test causing an unexpected cutoff was just a few months before the first flight.
This bug would have cost less than an hour had it been caught during review. It might have cost a day or two if it was caught during integration, causing a single test repeat. If it was caught during formal qualification, it could have cost a week or two by causing a complete test series restart with a new build.
As it was, the cost balooned. First we designed and ran tests to determine if the flight unit could even trigger the condition. After it was determined to be a real possibility, there was cost for engineering, management and customer analysis of the best fix, releasing the new build, creating and executing a new regression test plan, system testing in multiple units and simulators. All in all it cost thousands if not tens of thousands of man-hours. Plus the original 15 minutes to actually make the right code change.
Sadly, like many things, it depends.
If a dialog message is misspelled, it may be 'trivial' to fix (update string, rebuild/package, redeploy). Or if a layout needs updating, a modification to a .css file may be sufficient.
If the bug is that the output of a critical method that has a 100+ page spec and proof is wrong, then the investigation itself may take hours or days. This is what the old 'axiom' refers to, and what, among other things, TDD and agile is trying to avoid (fail early and clearly, make safe incremental progress, yada).
From my recent experience with multi-scrum teams on a single project, the 'bugs' are usually merge/integration issues that only appear at the close of a release as feature branches are promoted to stable. These are the worst, as the conflicts often require cross team support while the teams are in a flurry to finish their own objectives, but I don't know that they are more expensive than other bugs, as they occur when they occur: late in the release, but at the earliest time they can. That's what makes them the worst.
