Is DRY the enemy of software project management?

Question

One of the most basic and widely accepted principles of software development is DRY (don't repeat yourself). It is also clear that most software projects require some kind of management.

Now what are the tasks that are easy to manage (estimate, schedule, control)? Right, repetitive tasks, exactly the tasks that should be avoided according to DRY.

So from a project management perspective, it is great to solve a task by copying some existing code 100 times and make some minor adaptations to each copy, as required. At all times, you know exactly how much work you have done and how much is left. All managers will love you.

If instead, you apply the DRY principle and try to find an abstraction that more or less eliminates the duplicate code, things are different. Usually there are many possibilities, you have to make decisions, do research, be creative. You might come up with a better solution in shorter time, but you might also fail. Most of the time, you cannot really say how much work is left. You are a project manager's worst nightmare.

Of course I am exaggerating, but there is obviously a dilemma. My questions are: What are criteria to decide if a developer is overdoing DRY? How can we find a good compromise? Or is there a way to completely overcome this dilemma, not just by finding a compromise?

Note: This question is based on the same idea as my previous one, Amount of routine work in software development and its effect on estimation, but I think it makes my point clearer, so sorry for repeating myself :).

Answer 1

You seem to assume the primary objective of project management is to produce exact estimates. This is not the case. The primary objective of project management is the same as for developers: To deliver value for the product owner.

A product using a lot of slow manual processes rather than automation might in theory be easier to estimate (although I doubt it), but it does not provide value-for-money to the customer, so it is simply bad project management. There is no dilemma.

It is well known that estimation of software projects is hard, and numerous books have been written and various processes have been developed to manage it.

If the only objective of the PM was to produce exact estimates, then it would be easy. Just pad the estimates to 10X, and let the developers play games for the rest if they finish early. This would actually be better than your suggestion to use copy-paste busywork to pad the time, since playing games will not reduce the maintainability of the product.

But in reality, the product owner want useful estimates and a quality product delivered as quickly and cheaply as possible. These are the actual constraints a PM will have to navigate.

In any case, I dispute your assumption that repetitive manual work is more predictable than automated. All experience shows that repetitive manual work is more prone to errors. And what if a bug is discovered in the copy-pasted code? Suddenly the cost of fixing a bug is multiplied with the amount of repetition, which makes the uncertainty explode.

Answer 2

You are right - copy-paste works great, and DRY has no point when your task is to produce a program for which either the copied template or the copy will not have to be maintained or evolved in the future. When those two software components have a completely different life cycle, then coupling them together by refactoring common code into a common lib which is itself under heavy development can indeed have unpredictable effects for the effort. On the other hand, when copying code sections inside one program or program system, all these parts will typically have the same life cycle. I will illustrate below what this means to DRY and project management.

Seriously, there are lot of such programs out there: for example, the computer game industry produces lots of programs which have to be maintained over a short period of some months or a year at maximum, and when that time is over, copy-pasting old code from a previous game where the maintenance period is exceeeded, into a new game's code base is perfectly fine and might speed things up.

Unfortunately, the life cycle of the most programs I had to deal with over the last years is very different from that. 98% of the requirements or bug fix requests which arrived me were change requests for existing programs. And whenever you need to change something in an existing piece of software, "project management" or planning works best when your testing and debugging efforts are quite low - which will not be the case if you change something in one place, but due to copy-pasted business logic you easily forget you need to change a dozen other places in the code base as well. And even if you manage to find all those places, the time to change them all (and test the changes) is probably much higher as if you have only one place to change. So even you could make an accurate estimation for the change, having the costs a dozen times higher than it needs to be can easily collide with the project's budget.

TLDR - whenever you develop a program where is no necessity or responsibility for bugfixing and maintenance of the original or the copy, feel free to copy. But if you, your team or your company is or might become responsible, apply DRY whenever you can.

Example

As an addendum, let me explain what "bugfixing and maintenance" means, and how this leads to unpredictability in planning, especially inside one product, by a real-world example. I have indeed seen these kinds of things happen in reality, probably not with 100 instances, but the problems can even start when you have just one duplicate instance.

The task: create 100 different reports for an application, each report looking very similar, some requirement differences between the reports, some different logic, but all in all, not many differences.

The dev who gets this task creates the first one (lets say it takes 3 days), after some changes or minor bug fixes due to QA and customer inspection its finished, it seems to run fine. Then he started to create the next report by copy-pasting and modifying the whole thing, then the next, and for each new report he needs ~1 day in average. Very predictable, at a first glance ...

Now, after the 100 reports are "ready", the program goes to real production, and some problems occur which were overlooked during QA. Maybe there are performance problems, maybe the reports crash on a regular basis, maybe other things do not work as intended. Now, when the DRY principle had been applied, 90% of those problems could be solved by changing the code base in one place. But due to the copy-paste approach, the problem has to be solved 100 times instead of once. And due to the changes already applied from one report to another, the dev cannot quickly copy-paste the fix for the first report to the other 99. He has to look into all the 100 reports, read them, translate the change to the modified report, test it, and maybe debug each of it individually. For the PM, this starts getting really hard - he can of course take the time for a "regular" bug fix (lets say, 3 hours) and multiply this by 100, but actually, this is most probably a wrong estimation, some of the fixes might be easier to make than others, others might be harder. And even if this estimation is correct, having the debugging costs 100 times as high as they needed to be will cost you company a lot of money.

The same will happen the next time when the customer asks for changing the color of his company emblem in all those reports, for making the page size configurable, or by some other new requirement which affects all reports in a similar manner. So if that happens, you can make an estimation for the costs and bill the customer 100 times the price he would have to pay when the code had been DRY. However, try this a few times and then the customer will cancel the project because he will probably not be willing to pay your exorbitant evolvement costs. And maybe at that point someone will ask the question why this happened and point with the finger at the person who made the decision for this copy-paste programming.

My point is: when you produce software for others, you have always at least for a short period of time the responsibility of making the thing work, fixing bugs, adapt the program to changing requirements etc. Even in a green-field project, these parts can quickly add up to far more than initially planned development effort. And especially when all your copy-pasted code is inside one product, the period of time of responsibility is for all parts the same, which is quite different from the situation where you copy-pasted some older code from a dead project which is no longer under active maintenance.

Answer 3

So from a project management perspective, it is great to solve a task by copying some existing code 100 times and make some minor adaptations to each copy, as required. At all times, you know exactly how much work you have done and how much is left. All managers will love you.

Your base assertion is incorrect.

The thing that makes software different from other professions is that you're making something new every day. After all, no customer is going to pay you to build something that someone else has already made. Project managers might like predictability, but their bosses like value. If you're just copy-pasting code with slight variations, you're not providing much value for the company.

Eventually the company will realize they can do the same work in a fraction of the time by hiring a good programmer. And if they don't, their competitors will.

Answer 4

Cut-and-paste programming eventually leads to abandoned software. I was a contractor on a system for ordering wireline services from a very large phone company. The system was cut-and-pasted ad nauseum because all the testing was manual and they didn't want to change any working code. The tiniest enhancement could result in a new copy of hundreds of lines of code. Originally the application was written to handle accounts of up to twelve physical lines. Of course this limitation was made in hundreds of locations in the code. After about four years business asked the team what it would take to handle larger accounts. They estimated about $18 million. At that point the project was turned over to an offshore team for minimal maintenance. The existing team was all laid off.

Organizations that think this way are getting crushed by companies with better technology.

Answer 5

An oft forgotten maxim that applies here is the rule of 3. This states, that it is OK to copy code once, but beyond that it should be replaced by generic code.

3 might seem like an arbitrary number but a common scenario is where data and logic are duplicated in an application and database. A example often cited is where there is a lookup table in the database and an enumeration client side. The difference in paradigms don't allow this to be readily stored in a single place and so the information often appears in both places.

Whilst it is nice to have DRY code, there can be times when business logic dictates an exception and so you have to create two or more bits of code from source that was generic before.

So - what to do? Code for the status quo (after all, YAGNI). Whilst code should be written for ease of modification, writing a whole raft of bells and whistles for something that might not be needed is just torching money.

Answer 6

In your question, you only list three functions of project management - estimate, schedule, and control. Project management is about achieving goals within the constraints of the project. The methods used to achieve goals within the constraints of a project are different for software projects than many other types of projects. For example, you want manufacturing processes to be highly repeatable and well-understood. However, software development is mostly knowledge work - it's non-routine and requires thinking rather than following rigid instructions and procedures. The techniques used to initiate, plan, execute, monitor and control, and close a software project are going to need to account for the type of work that needs to be done on a software project - specifically, non-routine work that cannot be done to specific instructions and procedures.

I think the other problem is that you are taking DRY, a concept that relates to repetition of information, and trying to apply it to managing tasks. DRY simply says that you should only have one authoritative representation of information. Project managers should be embracing this, since it means that everyone will know where to go for the information, communicating changes will be easy, and the changes can be controlled and managed well. DRY, through reusable pieces, helps to keep long-term costs down, helps to maintain long-term schedules, and to improve quality - three pieces to the Project Management Triangle. There needs to be some investment of time and money spent to effectively make things DRY, but the job of the project manager is to make trade-offs of time, cost, schedule, and quality.

Answer 7

Writing new code is just a small part of the task

Your suggestion would make it easier to estimate the part of initially writing new code. However, for actually bringing anything new (no matter if it's a brand new system, an addition of feature or a change of functionality) doing this is not sufficient and is just a minority of work - the estimates seen in literature tell that in practice this part is something like 20%-40% of the total work.

So the majority of work (which includes adapting your initial development to what was actually needed, integration, testing, rewriting, re-testing) doesn't get easier to estimate; just the other way around, intentionally avoiding DRY just made that part much larger, harder and with more variable estimates - that bug or need-for-change which requires changing all the cloned parts may not happen, but if it does, then your estimates are going to be totally wrong.

You don't get better estimates by improving your estimation quality of a small part of work but making it worse on a large part of work; so it's not really a tradeoff but a lose-lose situation where you get worse productivity but also worse estimates.

Answer 8

DRY is useful but it's also over-rated. Some people can take it too far. What many developers fail to realize is that anytime you implement DRY to use the same method for two (slightly) different purposes, you are introducing a kind of very tight coupling between the different uses. Now every time that you change the code for the first use case, you have to also check whether it regresses the second use case. If these are broadly independent use cases the it's very questionable whether they should be tightly coupled - they probably should not be.

Over-use of DRY can also lead to God-methods that explode in complexity to handle all the different use cases to which they are put, when generally smaller atomic methods that replicate some code would be much more maintainable.

However, I would suggest that the question isn't really relevant at a project management level. A project manager really won't want to concern themselves with this level of implementation detail. If they are it's probably micro-managing. Really... how things are implemented is more the responsibility of the developer and technical lead. Project management is more concerned with what gets done and when.

EDIT: per comment, I do agree however that insofar as making it easier to estimate development time, avoiding DRY can sometimes reduce the amount of uncertainty. But I believe this is an insignificant issue in relation to the more pressing questions of (1) how long until business requirements are met, (2) what technical debt is taken on board in the process, and (3) risks to total cost of ownership of the architectural choices made - whether to go DRY or not in many cases is a design choice that should be based more on risk/reward to those factors, than to whether it makes it a little easier to provide project managers with more accurate information.

Answer 9

I think you are misunderstanding DRY.

Let's use an example:

public Class A
{
    public int Multiply(int x, int y)
    {
        return x * y;
    }
}

public Class B
{
    public int Multiply(int x, int y)
    {
        return x * y;
    }

    public int Add(int x, int y)
    {
        return x + y;
    }
}

vs.

public Class C : A
{
    public int Add(int x, int y)
    {
        return x + y;
    }
}

By replacing class B with C we have followed the DRY principle and reduced code duplication. But we haven't increased the unknowns or risk to the project (unless you have never done inheritance before).

I think what you mean when you talk about DRY is something more like a design task. I.e.:

public Class A
{
    public int Multiply(int x, int y)
    {
        return x * y;
    }
}

!!!New requirement! Some clients need to be able to multiply doubles!!

// Use class B for new clients!!
public Class B
{
    public int Multiply(double x, double y)
    {
        return x * y;
    }
}

vs.

public Class A // Version 2
{
    public int Multiply(int x, int y)
    {
        return Multiply(x as double, y as double);
    }

    public int Multiply(double x, double y)
    {
        return x * y;
    }
}

Here (assuming it works) we have designed a solution which can deal with both the old AND the new requirement, essentially trying to create a mathematical model of the real life problem or business rules. In real life the system we are modeling will obviously be much more complicated, our model of it won't fit exactly, and the edge cases and unexpected results will take time to find and correct.

So, should we take B or A version 2 in this case?

• B is going to be more specific to the actual requested change with fewer side effects and be easier to estimate and quicker to do.

• A version 2 will result in less overall code going forward and will be the more elegant solution

Again I'm going to say it comes down to the quality of the specification and requirements.

If we have very clear specifications which cover the edge cases and backwards compatibility then we can be sure we understand the system well enough to refactor the model without producing bugs.

If we have an emergency request for a single customer where the only requirement is that behaviour changes for that customer without consideration of the overall system; then 'improving' the model by refactoring A carries a substantial risk. Both of breaking other customers or overrunning the deadline because of the extra unknown time needed to design and test the solution.

Answer 10

Paragraph by paragraph

One of the most basic and widely accepted principles of software development is DRY (don't repeat yourself). It is also clear that most software projects require some kind of management.

Correct.

Now what are the tasks that are easy to manage (estimate, schedule, control)? Right, repetitive tasks, exactly the tasks that should be avoided according to DRY.

Repetitive tasks should be automated, mandatory. They are boring, error prone, when made by hand.

So from a project management perspective, it is great to solve a task by copying some existing code 100 times and make some minor adaptations to each copy, as required. At all times, you know exactly how much work you have done and how much is left. All managers will love you.

I think you can change the word "adaptation" with "configuration". Consider you have a bug in this piece of code that is supposed to be copied. A bug that appears under specific conditions. If it is not fixed in the original source, and it is copied there will be a lot of places to fixed. This may be bad, but then someone has to:

• first fix the code in the original source;
• fix the code in every other place;
• make sure that those were all of the places. When you say this had to be done to the manager, he will probably at least hate someone.

If instead, you apply the DRY principle and try to find an abstraction that more or less eliminates the duplicate code, things are different. Usually there are many possibilities, you have to make decisions, do research, be creative. You might come up with a better solution in shorter time, but you might also fail. Most of the time, you cannot really say how much work is left. You are a project manager's worst nightmare.

Removing duplications leads to single point of failure. If something fails you can be pretty sure where this happens. S.O.L.I.D. and Design Patterns are there to help to fix exactly that problem. Too short deadlines tend to provoke procedural style "coding". More time invested in one project in order to create something reusable means there should be minimal amount of time spent in the next project when the feature will be reused, but it should be configurable in first place.

Of course I am exaggerating, but there is obviously a dilemma. My questions are: What are criteria to decide if a developer is overdoing DRY? How can we find a good compromise? Or is there a way to completely overcome this dilemma, not just by finding a compromise?

A lot of people pointed there is no dilemma here. Yes and no.

If you have something highly experimental that has never before being done - there is no dilemma. Otherwise if you have something that has to be done again, like new booking system you already have abstractions, just depends what you need.

I think the dilemma is - should we implement a something in a feature, if it is unlikely to be requested. Implement something when requested. Nobody needs a huge infrastructure that won't be used.

Answer 11

this is not at all about designing for future reuse or about the YAGNI principle. It is about repeating code in the current work package.

It absolutely is about design. Maybe not re-use per se, but design nonetheless.

What are criteria to decide if a developer is overdoing DRY?

Experience and your existing environment/situation. For a given problem you will get a strong sense of the Prado Principle as you attempt greater degrees of DRYness. Then suddenly management considerations come to play. Time, goals, the customer, long term code management (someone said technical debt), etc. will inform your plan of attack.

How can we find a good compromise?

Uh... design? Refactoring is design, well it is supposed to be. The scope of the DRYing can easily expand like a super nova from the loop, to method, to class(es). Been there, done that. But you can't really know until you study the problem - this is design.

How can it not be a design problem? You must consider the problem more widely than the immediate duplicated code at hand. This is a design activity whether it's existing code or a blank sheet; whether it's "extract method" or creating new classes and modules.

Epilogue

... the referred question and its answers do not cover the project management aspect.

Typical management, ignoring design time. We would have, ideally, designed out the superfluously redundant repetitiveness prior to coding. Instead, management thinks development (and bug fixes) is a single Olympic event - coding - when it is in fact a decathlon. And they measure to 1/1000 of a second because they think it's all analog.

If instead, you apply the DRY principle and try to find an abstraction that more or less eliminates the duplicate code, things are different.

I had this experience: "I spent two days writing this row (of a GUI form) and two hours writing the rest of the form." I mean that I took the time to identify reusable classes - DRY being a natural side effect- the GUI form row and w/in that some others. Once debugged these were used, individually and in composition, throughout the form which now coded very fast and testing was exceptionally quick despite building-up complexity. And it flew through formal testing too with amazingly low bug rate.

Most of the time, you cannot really say how much work is left. You are a project manager's worst nightmare.

I didn't know either but I had faith the up front design effort would pay off. We all say this but management in particular doesn't trust it. Management would have thought I was screwing around. "Two days and you don't even have 2% of it coded yet!"

In one case we stuck to our guns when management said "you're spending too much time in design, get going." And co-workers saying "it's too many classes." Well, a far less complex sub-project was supposed to take about 1 month (I thought it was OK ballpark guess) but took 5 months. 3 months of that was in testing/fixing because it was such a POS. "But we didn't have time to design!". They actually said that.

My questions are: What are criteria to decide if a developer is overdoing DRY? How can we find a good compromise? Or is there a way to completely overcome this dilemma, not just by finding a compromise?

Show management how it works out. Capture some data. Compare to other work, especially that of your co-workers who do the slap-dash rush job. That pile of failure always seems to lose the race, getting stuck in test and then after release returned over and over to fix more bugs.