Coping with an unfixable endless project

Question

We have a large (1200+ hours) website that has a lot of technical debt. This is mainly caused by the following (usual) reasons.

1. Multiple programmers who come and go during development.
2. Change of specifications during development.
3. Numerous added functionalities added (in a short time).

The customer wants alot of new functionalities, and that basically comes down to working on this project weekly for 10+ hours.

Due to the technical debt, we spend A LOT of hours fixing or investigating problems, that usually find their origin in one of the following:

1. A shameless, silly bug that makes people cry.
2. A new feature results in the above because we hadn't foreseen all the places the new feature would have an influence.
3. Some other problems we have faced (f.e. server migration, upgrades)

We have issues daily and we have tried to following things to put this to an halt:

1. Created technical documentation regarding the import, payment and general working of the website.
2. Have meeting at the start of the week - discussing the current issues or improvements and how they should be tackled.
3. Have a test-plan. Programmer A test B, B tests C and C tests A. Then our Project Manager will throw in some tests. Regarding the impact of the feature we throw it on a staging environment and let the customer check for itself.

The problem is that the problems keep happening...and somehow we can't get a grip on it. New features still cause bugs, and old bugs keep saying hello. Somehow - perhaps due the size of the project - we can't seem to get a grip on this project.

I assume there are alot of programmers working on larger projects then this. That is why I come to my question:

What can we do, or what do you do to avoid these problems on large projects?

Minor edit, extra info:

1. We use version control (SVN).
2. We have DTAP development process.

Answer 1

I'll play devil's advocate, having seen far too often how this turns out: You can't cope with it. I guarantee you're the only one who actually sees a real problem with the system as it is, or else you wouldn't have to ask how to cope with it because the company culture would be one to stamp out bugs and fix the code wherever possible i.e. operating how real professionals work.

I bet it's too large to start writing unit tests, because it hasn't had anyone who knows how to unit test before you (and with luck other people on your team) and it's impossible to know where to start, and maybe even impossible to test because it relies on exact implementations and concrete data, so it would take far too long to strip all of that out to interfaces, mocks, stubs and the like to be able to test it in the first place. I also bet you can't just go and refactor what needs to be refactored because it's too tightly coupled and, since there are no tests, who knows what will be broken by fixing bad code. In short, it's probably become too cancerous to seriously fix, but of course it can't just be cut out and start fresh.

You're fighting a losing battle, my friend. Either you'll burn out from frustration and eventually quit or go insane, or if you complain about it long enough trying to get others to realize the issues, they'll think the only issue is you and you'll be shown the door.

Answer 2

Unit-testing things is a good starting point if you're not doing any. At the very least they'll protect you from adding new bugs when fixing old bugs.

Source control also helps, unless you're not using it. The blame and log features, in particular, are wonderful to nail down how/why a buggy piece of code ever got committed.

On the customer's end, I've found that discussing price and (lengthy) delays as soon as changes/additional features are requested works reasonably well, as does charging for the time you spend discussing/designing them. Frequently, customers will decide that on second thought they can wait.

(By contrast, if you immediately dig into specs and implementation ideas with him, they'll typically set you up for a "oh, I thought we had agreed you'd do this anyway" or (worse, after several days of back and forth on the specifics) "but look, it's designed already and we what we discussed doesn't sound that hard!".)

Last but not least, I've found that being upfront that I only read emails once per day (upon arriving at work), and that I've a phone for anything more urgent, leads to a tremendous productivity increase.

Answer 3

I suggest you add some CI-based testing, primarly on the areas that break most frequently. That will help you increase quality as work is being done on the project.

It's also becomes more apparent which areas/functionality break more often and thus it's easier to decide which parts need refactoring, or at least increased testing.

Adding more manual testing risks having the project go the wrong way in terms of $$$ & time required per feature added.

Some code review is good to, but maybe that's part of the A->B->C->A testing scheme. (Maybe code review in the other direction?)

Answer 4

Let me throw a fable at you. You were taking a walk with a person earlier in the day down the street and you reach your destination. The person you are walking with quickly finds out that he lost his ring somewhere along the way so you both decide to backtrack and go searching for it. The person you are walking with quickly stops at a lamp post and begins looking frantically. You say, "Why are you looking there at the lamp post when I think you might have lost it when we cut through the alley?". He replies, "I know but the light is better here."

I have been in this situation more than a few times and I have noticed some commonalities. These kinds of maintenance nightmare projects are typically run in a process heavy environment with heavy oversight and process improvements imposed by management. I am not saying process improvements are a bad thing but more often than not the types of process improvements that Management will typically want to enact have two key points.

1) They generally don't disrupt the office politics and balance of power. 2) They are successful at creating the illusion of control by management rather than strike at the heart of the issue.

The "light is better here" management think typically goes about by saying, "Every new feature must have a detailed tech spec", or "Lets have an hourly status meeting everyday to discuss issues and how to overcome them."

Neither of these things really strike at the heart of the issues and they might just decrease productivity but they certainly validate the illusion of control by management.

The only true changes that you can help push for would be ones that shake things up. I suspect though that your monstrosity of a web site is probably beyond repair at this point and you would be further ahead to re-architect and rewrite. For the future however you can keep in mind the importance of Agile methodology, Continuous Integration, Test-Driven Development, Code-Reviews, and business requirement specifications that are regulated under strict Change Control procedures to help minimize scope creep without schedule adjustments.

These kinds of changes truly require a change in the way of thinking at the management level and in my entire professional experience I have never encountered this to happen without some kind of middle management level shakeup. I hope that isn't too discouraging as you should try for what is right regardless if you are fighting an uphill battle, because you will likely encounter fierce resistance by people who love the status-quo.

Answer 5

I've been in the same spot some time ago. I'm not anymore thanks to two simple rules:

• Every week one, or two days are spent fixing/rewriting most hairy parts of the app. No bug hunting, no new feature development.
• While implementing new features we strive to get it right even when we spent more time than customer is expecting.

The only problem is to get other people to respect them. The easy part surprisingly was customer. Can't really explain why, but somehow we have convinced him, that when we work on a feature a bit longer it's better for everybody. Respecting the first rule turns out to be more problematic, but also we feel it helps us a lot. It guarantees steady progress as different parts of application are getting better.

Answer 6

Code reviews. Unit tests. Real QA Testing. Specification collection process and incremental development - these are some things that should resolve most of your issues.

Also don't let the customers directly ping your developers - This is usually the most unproductive way of solving issues. Hire a good Program manager who will form the interface between your customers and developers. His job would be to know the product end to end, current status, future directions and so on. Any time the customer wants another new feature, he should be able to give the current list of items and show the customers what will get bumped off if this new request is to be taken.

Process is bad when it is used too little or too much. I think you are at the former state.

Answer 7

As Deni mentions, if you are able to add unit testing to the project, then this would help you. Have a test which covers a part of the system your about to change/fix, and so when your refactoring code, use that test as a guide to make sure your not breaking anything.

Also, triage the most broken parts of the code. Try to put those worst affected into a list of risks and manage those risks independently. Try to get an idea of how much broken code there is in the codebase by querying where the bugs happen most. You can then list the affected area by the bug count (or issues reported, whatever works for you.).

Patching up and cleaning the code will take time, but if each dev on the team can leave the code a little bit cleaner then it was before they toched it, then over time the codebase will improve. If your looking for a quick, army style, get it sorted now solution, I'm doubt that there is anything practical (or recommended) which would help.

Cheers. Jas.

Answer 8

Write clear functional specifications; pedantically so if you can bear it and review functionality against those specs regularly. The less of an idea a dev has about what he is supposed to be developing, the less chance there is of it being the way it is supposed to develop.

Before you start writing code do some up front design work; this doesn't need to be perfect, or huge, or contain UML, but it should outline a fairly solid solution to the problem that needs to be solved. As far as I can tell the less software is planned, the worse it is. Discuss the design before you start working on it.

When you start working on an area of the code that is clearly bad and really hindering your progress; stop adding to it, step back from the problem, work out how you could redesign the architecture so that the hindrances weren't there and so that it would be more adaptable in the future. The longer you leave it before you address technical debt the harder it will be to address it without a full rewrite. I'd say it's an exponential thing.

Design tests that test behaviour and don't couple tightly to your architecture. It isn't very fashionable, but I would say don't start testing until the true purpose of your code is clear. Specifically don't start testing until you know what you really want to test; IMO a poorly thought out test is worse than no test. And the more tests you have the harder it is to change your code internally. Treat your test code as you would production code; it needs to be planned and well written.

Do regular / daily code reviews: this is more about sanity checking to make sure the dev hasn't gone to far off course. Use these sessions to plan out the next days work. There may be days when these take 5 minutes, or 1 hour; the point is to keep a dialogue open and give developers a chance to discuss their work with other developers and seek advice. Do some pairing sessions on tough parts of the code, or to prototype ideas, but let people have their own time to work.

Make it easy to build and deploy your code. Try to keep build times short. The easier it is to build the more it will be built, the quicker it is the more it will be built.

Adopt coding standards and enforce them rigidly. This should cover everything from where a project should live in the file system to the casing of a private const. This might seem pointless and annoying, but good habits are the cornerstone of a development process.

Fundamentally I don't think that the process you use matters so much, fashions come and go. What really matters is that you are professional about how you develop software and that you are disciplined in your practice.

Answer 9

I'd start off by designing and automating smoke tests, and throwing them into CI environment. Those should be functional. When the customer tells you that something should work so-and-so, ask to write it down, so that you may refer to it later. When you see a certain solution in the software, ask questions, and as soon, as you receive answers, incorporate them into knowledge base, and make them traceable.

Assure, that the basic functionality for positive cases works. Then start building incrementally tests for incorrect data handling, placing defects where deemed necessary. Have a long and deep discussion about priorities, and have the test manager aware thereof, so he can assign testing time accordingly. Don't attempt to automate everything, but as soon, as some testcases make sense to being automated - don't hesitate.

In general, use testing for increasing trust in the product, and not as the tool that will instantly increase quality. Be calm, yet assertive :). Maybe attempt going agile, but only if you absolutely, positively may engage a certified PM. Introducing Agile by a person who don't know Agile, will most probably kill the project.