5

I was arguing this point with my boss the other day. He claims that the larger the code base the riskier the deployment.

I argued that this is not true, though I see why he might think that. In my experience the size of the code base doesn't matter if you follow solid principles and have good tests.

I tried to give examples of modern cars and how reliable they are even though there are thousands of moving pieces. This is because each piece does its job and only its job and it does it well.

Am I crazy, or is this just one of those counter intuitive things that some people have trouble accepting?

Improve this question
8
  • 5
    Does the likelihood that every component was developed with solid principles and good tests increase or decrease with the number of components?
    – Jacob Raihle
    Dec 5, 2019 at 11:15
  • 2
    See Greg Wilson - What We Actually Know About Software Development, and Why We Believe It’s True. In particular from minute 39 second 30. However, I still recommend the whole presentation.
    – Theraot
    Dec 5, 2019 at 12:10
  • 2
    This sounds like one of those arguments that everyone and no one can win because "riskier" is a word that everyone thinks they agree on but really don't. In other words, when you say "riskier" what are you actually talking about ? Security? reliability? updatability? etc? You can make arguments for each one individually (and some in concert) but not all of them as a whole.
    – Matthew
    Dec 5, 2019 at 12:57
  • 1
    What, exactly, do you mean by "riskier the deployment"? Do you mean the risk that you build the right thing? Risk in number of defects? Risk that the deployment will, for some reason fail? Something else?
    – Thomas Owens ♦
    Dec 5, 2019 at 13:45
  • 4
    @Theraot that video is interesting, thanks! For the lazy: the speaker references a study that shows that no metric predicts flaws in code better than simply counting lines. So lack/presence of comments, descriptive function names, modularization, etc, none are better correlated with bugs than simply doing "wc -l your_source_file". And this is backed by statistics!
    – Andres F.
    Dec 5, 2019 at 15:24

5 Answers 5

Reset to default
7

When a typical software product grows in size over years, there will be definitely more places where bugs can hide, and occasions where more complex, unforeseen interactions can happen - this is where your boss is right.

On the other hand, countermeasures against degrading quality involve often even more code in form of

  • automated tests,

  • automated deployments,

  • more in-product validations,

  • improved configuration management,

  • code generators

  • redundant functionality

  • additional security layers

and so on. And you it also requires process improvements like code reviews, manual tests, documentation etc.

So the risk of a deployment (whatever that means) will to some degree depend on the size of the codebase, but not alone - it depends also on if your system has found the right balance between code which is necessary for certain features and code for supporting those other measures to keep the overall quality high.

"Following solid principles and have good tests" - as you wrote - is a good start, but far from being sufficient in the long run. The more features a system contains, the more important it will become to apply all known means of quality assurance and quality management.

So in short: both of the two perspectives you described contain some truth, but both look to me as if they are oversimplifying things.

Improve this answer
9
  • "The larger a system gets, the more important it will become to apply all known means of quality assurance and quality management." I would say this supports the assertion that larger systems are riskier, so the manager is right :) edit: in fact, I'd say your whole answer leans to the manager's side!
    – Andres F.
    Dec 5, 2019 at 15:07
  • @AndresF.: they are both right and both wrong. But I changed my wording a bit, maybe that makes it clearer.
    – Doc Brown
    Dec 5, 2019 at 15:08
  • Of course, you know best what you meant, but my reading of your answer makes me think you're saying the manager is more right ;) "Does not simply depend on the size", for example, implies it does depend on the size -- just that it's not the only dependency.
    – Andres F.
    Dec 5, 2019 at 15:11
  • @AndresF.: I definitely disagree with the statement in the question "the size of the code base doesn't matter", at least literally, even though it is accompanied by "if you follow solid principles and have good tests."
    – Doc Brown
    Dec 5, 2019 at 15:14
  • 1
    On the part where the boss is right, I wouldn't use a "codebase size" metric, but rather a "how many things have been altered since the last succesful release" metric. Seems to be more correct, no?
    – Flater
    Dec 5, 2019 at 23:10
2

Let's say you have a 1% chance of any given module having a critical bug. If you have two modules that both must work in order for your system to work, you now have a 1.99% chance of your system failing. If you have three such modules, you now have a 2.97% chance of your system failing. That's basic reliability engineering.

I oversimplified, but the main takeaway is that the reliability of the system is worse than the reliability of each individual critical component, and gets worse with each additional critical component. If you want to increase the reliability of a system, you must specifically design for a failure of one component to not affect the system as a whole. For example, by running redundant services on physically separate machines.

Improve this answer
2

The Example is poor

A Modern car is complicated but highly reliable for the same reason that Lego can make complicated structures but still hold together reliably.

They are both restricted to 4D interactions.

There are of course a few more principles at play such as:

  • Well defined interfaces
  • Mass Produced
  • Quality Assurance of components

Compare that to software code. The 3rd, 400th, and possibly all lines between 2001- 69876 can influence variable A which then influences...

The dimensionality boggles the mind.

Now if you write code and restrict yourself to the low tens of dimensions, and implement other principles like: well defined interfaces, mass production, and quality assurance you can indeed approach levels of reliability found in cars.

In fact these systems exist, such as the flight avionics in planes. These are modular with well defined interfaces, each module is itself redundant with each duplicate produced independently by competing teams, and quality assurance is thrown at every component and assembly.

Risk

Your boss is correct in assuming that a larger code base is riskier, because that extra line can push the dimensionality of software higher. Higher dimensionality makes a larger result space that must be quality assured.

As there is only so much time in the world, only so much of that result space can be verified. The greater that difference the riskier that code base is, because it is less likely day to day operations will stay in the verified region.

You are correct in that applying good engineering practices allows you to reduce the internal dimensionality and as a result the size of the result space. It also allows you to verify regions of the result space much more reliably and in a more timely fashion.

I think you can also agree that the most reliable code, is the code that does not exist.

Improve this answer
1

With two otherwise equal programs the larger one obviously has more risk.

But when do you ever have two equally well programmed programs?

What you are attempting to argue is that your large program has been created smarter than the average program and lacks a proportional amount of booboos.

Which is probably not a convincing argument to a manager. I suggest you argue for measuring the risk by recording deployments, changes, bugs, roll backs etc

Improve this answer
2
  • I agree with your last statement and that is part of it. I dont think there is a cumulative effect in a robust mature design though and that is why I argue that size doesn't matter at least not nearly as much as other things
    – John S
    Dec 5, 2019 at 18:58
  • 1
    So you believe releasing two small programs is as risky as one small program?
    – Ewan
    Dec 5, 2019 at 19:09
1

The popularity of Microservices and the other answers here suggests that your boss is correct in "the typical case". Your argument seems to be that the typical case is not absolute and that exception software engineering can deliver extremely high-quality software in a single large code delivery.

If I were you boss I would smile and tell you that while I hope for exceptional software I plan on the occasional worst case of average software. If your boss wants you to spend more time breaking down your delivery into smaller incremental features to reduce risk then, well, they are the boss.

Your boss appears to be indicating that they want to see smaller releases or feature flags. You are then saying "Actually, no, as I am delivering superior engineering through A, B and C". This is probably not a great career move, as your boss is saying they think things are actually too risky, they want you to push out smaller releases, and you are making a general argument that they may be wrong in some cases.

The correct way to think about this scenario is not pondered "who has the correct view of software engineering?". The correct way to think about this is the cost/benefits to yourself. Consider that you take the advice or ignore it:

(A) you get to push out a big code drop, or

(B) you do as your boss says and break things up into smaller code drops

Then some possible outcomes at the next release:

  1. there is a bug that has a negative business outcome, which includes superficial reputation damage as well as actual financial damage.
  2. the code is great. no significant defects. no negative business outcomes.

There is likely to be no win for you if you ignore your boss:

  • {A,1} a big code drop and a negative business outcome. You ignored your boss and thing went bad. You should get fired.
  • {B,1} you did smaller changes. A subtle bug led to a really bad outcome. Your boss knows you did your best and took their advice. You don't get fired.
  • {B,2} you did as your boss asked. Nothing went bad. Win-Win.
  • {A,2} a big code drop and no negative business outcomes. Yet you have actually ignored your boss on matters of risk which is their responsibility. This is a Win-Lose. You are a gambler who now needs to win every time as the first time you slip up your boss is going to pound you.

My advice is that when it comes to risk assessments then it is the boss's view, where they may know of political risks they cannot share with you, that is paramount.

When I have been the boss in such cases where there I flag up risks and the developer pushes back it is usually because of developer fatigue:

Dev: "This code is ready to ship."

Me: "Hum, this looks risky, because of ..."

Dev: "Nah, I have had enough of this code and I want to ship it. Things will be okay because of X, Y, and maybe Z."

Me: "Let me tell you a story, I did this feature and things went bad because of ... but I didn't get fired as my boss and my team stood by me."

Dev: "Er. Well, if you really want me to do more work to {write tests, simply the code, add a feature toggle, break it up into smaller releases, change the error handling to fail-fast, use a library not my customer algorithm, do rigorous performance profiling ...} (delete as appropriate) then I can.

Me: "Thanks. I really appreciate it."

In general "I don't want to do more work" isn't really a great basis to argue with your boss ;-)

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.