In this question on the Aviation site, one of the reasons for the long development time of the F-35 was quoted as being Software. 8 million lines of code. Development from contract to in-service took 20 years, the last 10 years of which were mainly for systems since the first production aircraft was delivered in 2006.

It seems to me though that military software is like consumer software, in that it is built incrementally on top of existing functionality, and that development can be divided over many developers. Are there any limits to this, is there a sound barrier for software projects? I don't know if 8 million is an inordinate number of lines, but would this size be a reason per se to justify 10 years of development time?

What methods are there to limit development time in large software projects? Has there been a Henry Ford of software yet?

  • 1
    8M LoC is very big. There is bigger software, but very few. The more developers you have, the more communication is needed and the more it slows down; so there is a natural limit as to how many developers you can have on a project. Also, military software has much higher quality requirements than typical consumer software: you dont want that jet to trun towards the ground (or even space) and then not respond to steering commands. 10 years for initial development seems reasonable to me. – marstato Jun 20 '17 at 8:53
  • 3
    Read The Mythical Man Month. Despite its age, it is very actual – Basile Starynkevitch Jun 20 '17 at 12:43
  • 4
    "Has there been a Henry Ford of software" software development and assembly line manufacturing are not analogous. In fact, it's hard to find two things that are so dissimilar. In an assembly line, your goal is to make a lot of the same thing such that each one is indistinguishable from another. In software, you never want to make the same thing twice. Ever. If you do, you are 'doing it wrong'™. There are few ideas that have been more damaging to the software development practices than that of trying turn software development into an assembly line process. It's completely bonkers. – JimmyJames Jun 20 '17 at 16:39
  • 4
    How many man years of coding would generate 8 million lines... -- "The Mythical Man Month" states that 10 lines of code per developer per day is a good rule of thumb. This might not seem like much, and certainly wouldn't befit a prototype, but by the time you get through all of your meetings, documentation, testing, paperwork and refactoring/polish to get a stable, reliable program that meets all of its specified requirements, 10 SLOC per day per developer becomes quite realistic. – Robert Harvey Jun 20 '17 at 17:46
  • 1
    What methods do software engineers have to limit development time? The budget – Laiv Jun 20 '17 at 18:17

tl;dr summary: Software Engineering is a young discipline that is still figuring out how to build complex projects, and 8 MLOC is a very large and complex project.

Software Engineering is young. Very young. Like, two orders-of-magnitude younger than other engineering disciplines. Software Engineering started with the First NATO Software Engineering Conference in 1968, whereas e.g. Civil Engineering, Structural Engineering, and Mechanical Engineering started more or less with the building of the Great Pyramid … that's 60 years vs 4600 years of experience.

Or, to put it bluntly: we haven't figured it out yet.

There are a couple of things that make Software Engineering different from other engineering disciplines, most importantly the fact that software can be copied and re-used at no cost, and that software can be abstracted, parameterized, and generalized. What this means is that usually we only build things that nobody has ever built before, because if someone had built it before, we could duplicate that solution at no cost. This is different from e.g. structural engineering. You cannot just snap your fingers and clone a bridge, but you can do that with software.

So, other engineering disciplines have decades, centuries, even millenia of experience perfecting building (more or less) the same thing over and over again, whereas SE has only 6 decades of experience building completely new things every time.

Oh, and make no mistake: if you think comparing the F-35's software system to one of the Great Wonders Of The Ancient World is unfair to the herculean efforts by the ancient Egyptians, I would argue that the complexity of building a software system as sophisticated and advanced as the F-35's with its avionics, sensor fusion, and virtual reality environment (to name but a few) is comparable to the complexity of building something like the Great Pyramid.

The F-35 is actually not one plane but three. The F-35C has larger wings, so its aerodynamics are different. The F-35B is actually two completely different aircraft in one, one which generates lift with its wings and one which generates lift with its fan and engine nozzle. While this project saves cost and reduces complexity (at least that's the idea, there are people claiming otherwise) compared to developing and maintaining three different airplanes, it obviously adds complexity compared to each individual plane that would hypothetically have been developed instead, had they not chosen the "Joint" concept.

Add to that the fact this is a government project, where you have many potentially conflicting interests influencing the project that actually have nothing to do with the intrinsic functional or even non-functional requirements of the project.

8 MLOC is a large project. Think about the Linux kernel, its total size is about the same (~10+ MLOC at the moment, I think). But, roughly 90% of that codebase are device drivers (for example for dozens of graphics cards, dozens of SCSI controllers, obscure industrial input and output devices you have never heard of, …), so the actual Linux kernel is only about 1 MLOC. Of that, 90% is architecture-specific support code (for about 20 system architectures from no less than 3 PC architectures (x86 (32 bit), amd64 (64 bit), and x32 (amd64 in long mode, but with 32 bit pointers to save space)) to multiple different ARM, PowerPC, MIPS architectures to CPUs you have never even heard of). So, the real OS is only about 100 KLOC. Now, an OS is generally considered a complex system, and Linux is considered to be a large-ish OS. Well, the F-35's software system is about 100 times as large!

What methods are there to limit development time in large software projects?

That is essentially what Software Engineering is.

| improve this answer | |
  • Current Linux kernels, i.e. 4.10 or better, have more than twenty millions lines of source code. And recent GCC have about ten millions lines. Firefox has more than 20 millions lines of code. However, I guess that the F35 code is proprietary with many subcontractors (and each of them don't see all the code), and that probably makes a huge difference. – Basile Starynkevitch Jun 20 '17 at 12:38
  • 1
    "a young discipline that is still figuring out how to build complex projects" I don't agree with this. We do know how to build complex projects. Its just lots of people are not aware of this and use their own inexperience instead of what really works. – Euphoric Jun 20 '17 at 12:41
  • @Euphoric It's not something that can be taught. Extremely talented programmers can still get caught up in the details of design and many disagree on the proper way to establish a program structure much in the same way that there is no "best way" to design a skyscraper, and there are certainly far more "bad ways" than there are good ways. – Neil Jun 20 '17 at 13:03
  • @BasileStarynkevitch: Do you know whether the relative (i.e. 1%) or absolute (i.e. 100KLOC) numbers are still applicable? I can easily imagine the architecture and driver code base growing faster than the core. GCC is probably similar to Linux in that it supports something like 7 front ends and 50 back ends, and I wouldn't be surprised if that dominates the size in the same way as with Linux. FF is probably the best example, with a significant portion of that code dedicated to support bad decisions made in hastily written specs decades ago. – Jörg W Mittag Jun 20 '17 at 13:07
  • A little more than half of GCC code is common to all all front-ends and back-ends: it is the large middle-end of it. On gcc-melt.org/docum.html you could find slides I wrote about that. – Basile Starynkevitch Jun 20 '17 at 13:18

One thing to consider about this particular example is that it's not simply the size of the code base or the complexity of that code. A big factor here is the risk involved. Consider 20 meter beam about the width of a balance beam set up less than a meter above the ground. Most people who can walk would consider walking across that beam to be fairly easy. Now consider that same exact beam placed 20 meters above the ground (in a controlled environment i.e. no wind.) Now most people would consider this a much more difficult task and would take more care in walking across it.

The act of walking across the beam is no different in either case yet it's clearly not the same to ask someone to do one over the other. The difference is the consequences of error. The case you have chosen here is one where the risks are immense. 1. As I understand it, a fighter jet is unstable and the computer systems are required to keep it flying. 2. The radar, navigation, stealth, and targeting systems are crucial to making it adequate for battle e.g. it could be destroyed easily by enemies if the computer systems do not function perfectly.

Most software doesn't have this level of cost. We usually are walking across the beam that is low to the ground. If we fall off, it's of limited consequence. This project is one where people could die and wars could be lost if it's goofed. The level of rigor and review is pretty much maximal.

Overall I think the short answer is that software does today get developed much faster than it was in the past. I don't have data, but I've been around long enough to notice changes. There are various reasons for this including but not limited to: better programming languages, better development processes, better standards, and better project management. One of the major factors has been the industry abandoning management models that were not appropriate for software development.

| improve this answer | |
  • Very good point, a bug can be deadly instead of just a nuisance. – Koyovis Jun 21 '17 at 8:45

F-35's software size is not that large when compared with other systems.

The development process can make a huge difference.

Simply switching from waterfall and feature+integration branches to trunk based development (with a lot of test automation and a good CI system backing it up) can significantly reduce development time for very large/complex software development projects.

Such transition in a telecom equipment software (several times larger than F-35's) cut development time almost in half. Also reduced the overall number of checkins to about 30% - saving lots of resources previously wasted on the integration hell and on trying to stabilize those branches which weren't really what was shipped in the end.

| improve this answer | |

Not the answer you're looking for? Browse other questions tagged or ask your own question.