How do professional software development teams deal with design complexity in non-trivial projects?

Question

First, I realize this question may by somewhat long and vague and I apologize for this. This is probably a basic problem with a short name to anybody who's "got it", but as I find myself lacking in this regard, please bear with me in describing the problem.

I've been doing programming in this way or the other since I was about 11 years old. This means that I've been mostly teaching myself everything from the start. I received a technical education, but not strictly in Computer Science (I graduated with a degree in Photonic Engineering). We had programming courses of course, but this was mostly basic stuff for me and I didn't learn much new things. I've kept educating myself along the way for the joy of it and always knew I would pursue a career in programming, but all my projects were quite small at that time. I had no trouble keeping them in my mind and maintaining them.

Now, I find myself the lead on a team, but not in a corporate environment - I work for the university developing scientific software (in C++) for engineering applications. Suddenly the project is growing (relatively) big and I've trouble wrapping my mind around it most of the time. I'm losing a lot of time and effort on two things mostly:

1. When I have to return to a section of code I've not worked on for a while, I have difficulty remembering how it worked. I spend a lot of time reviewing the header files for the relevant classes and reading the comments I placed along the way in the source files. I wish there was some form of "schematic" I could glimpse at and regain the picture more easily;
2. When I introduce changes, sometimes I realize half-way that what I'm trying to do will break things somewhere else (or worse, it shows up only at runtime as a surprise). I revert and start doing it differently, only to find out I neglected the influence on some other component. I wish there was some "architecture diagram" where I could see how things get done, how what I'm trying to do will influence other components and a way for me to plan in detail before I start implementing changes.

Most of the people I work with have similar stories as my own - strong technical orientation and sometimes great skills, but with no way of organizing their work. However, their projects are usually much smaller than mine so they cope somehow. Anyway, what that means for me is that I'm on my own and I have nobody to learn the good practices from.

I took up a postgraduate course in managing IT and while I find it quite satisfying, it's mostly targeted at non-programmers, teaching about project management methodologies, budget/schedule estimations, enterprise architecture etc. - not software design and planning as such. That's OK, I'm trying to learn that stuff too. Of course, some tools (like UML) and the types of sofware development processes (cascade, iterative, agile...) were introduced, but obviously not in great detail and I've a hard time deciding what I should pick and use (and to what extent).

I've been reading many questions and answers about software design on SO - there are many about doing it using this or that particular tool or methodology, and if I was convinced that UML documentation would solve my problems - I'd pick it up and start using it. But some people swear by it, others say it's useless. I'm looking for an answer on a higher level of abstraction - are there ways to solve the two problems I'm having, and how do you personally do it? What should I learn to be able to do it, possibly without being bound to one particular tool? These come and go out of style from time to time, and I expect their applicability varies depending on the type of the project.

Thanks a lot for reading, I was unable to say what I mean more briefly (lacking in software design experience and vocabulary).

Answer 1

When I have to return to a section of code I've not worked on for a while, I have difficulty remembering how it worked. I spend a lot of time reviewing the header files for the relevant classes and reading the comments I placed along the way in the source files.

Imagine how the poor guy who comes after you is going to feel -- he doesn't even have the benefit of having once known how your code works. Instead of trying to decipher your code, you should be reviewing the documentation that you wrote for the module in question. That documentation should offer a reasonably accurate view of what the module does any why. It's not "the module starts by initializing three arrays using a triple for loop...", but instead: "this module retrieves the data collected by the Fabulotron's main sensor, rearranges it into standard Neopolitan (chocolate, vanilla, strawberry) format, and delivers it to the Analysis module."

In a perfect world, you'd have a design document that sets out the various modules in the system and describes their respective responsibilities, and each of your modules could just refer back to that document to explain what they do: "This module provides the Fabulotron data collection service as detailed in section 4.8 of the design document: http://fabulotron.org/design/section4-8.html." If you don't have something like that, start writing down an overview of each module as you work on it. You don't need to write a book -- a few paragraphs are often enough to get you oriented.

When I introduce changes, sometimes I realize half-way that what I'm trying to do will break things somewhere else (or worse, it shows up only at runtime as a surprise). I revert and start doing it differently, only to find out I neglected the influence on some other component.

That might be an indication that your modules are too interconnected. The more independent you can make your modules/classes/units, the less likely it will be that you'll run into this kind of problem. Try to make the interfaces between modules as explicit as possible, and try to limit them to just what needs to be there. An interface is a contract -- if you know that some module lives up to its obligations as specified in the interface, you don't need to know anything else about it. That prevents changes in the module you're working on from affecting other modules.

I wish there was some "architecture diagram" where I could see how things get done

Using standard parts can help in this respect. C++ provides standard parts in the form of the Standard Template Library, and using those parts where appropriate lets you work at a higher level of abstraction. If you've written your own code to manage data structures like lists, you and those that follow you will constantly have to read the source to figure out what's going on. If you instead use standard parts provided by the STL, any C++ programmer will quickly be able to tell what your code is doing without digging into your data management routines.

Another kind of standard part comes from design patterns. They're standard concepts that can be used as a shorthand to explain how the relationship between two objects works.

Answer 2

I have only been a professional developer for a short period of time, and struggled a lot with how to do this when I first started. I was primarily self educated, even through University. Fortunately for me the people I worked with have a lot of experience and were able to educate me on the ways to manage and work on large projects.

One of the first things that had me do was to sit down and read clean code. Which is a fantastic book that helped me to understand how to write code that I could understand when I went back to it or that someone else could understand.

The second thing was to write good quality tests, Unit, Integration, Acceptance. If your code is well tested you'll quickly know when you've broken something and quickly be able to diagnose the problem. There are plenty of good testing resources out there on the internet, and I'm not sure which are the best to suggest to you.

My key points are Testing and Clean Code.

Answer 3

Here are some high-level ideas for organizing large software systems. Most of my experience is in internet systems, but I think these ideas apply to engineering software.

• Separate functionality into relatively isolated modular blocks. For example, you might have a module for each family of calculations your software provides.
• Apply the MVC design pattern for the user interface, if there is one. Keep the interface and the model separated with well-defined boundaries.
• Consider using layers to group modules. In an application that uses abstraction layers, each layer depends only on the layer below it.
• When writing documentation, assume you will forget all of the implementation details. You will write a lot of documentation under this assumption - maybe as much as half of your code will be comments, but you'll be much less confused later.
• Automated unit, integration and acceptance tests are great in some spaces, but C++ developers seem to have mixed feelings about them.
• Draw heavily on design patterns. Besides being generally good ideas, design patterns provide a common vocabulary in software engineering. Calling a class a WidgetFactory is a concise way to communicate that it is a class that exists to make widgets.

It's been a long time since I worked with engineering software, so your mileage may vary on these suggestions. Good luck!

Answer 4

The answer is software engineering.

That is for large projects you follow a set sequence of requirements gathering, process definition, specification, etc. long before any actual coding is done.

There are various methodologies used depending on the environment and culture. Enterprise type business tend to follow "Rational Unified Process" or similar, tech start ups usually go for some variation on Agile, government departments some overworked Waterfall methodoligy.

In all cases the process helps you define exactly what you are doing, and, towards the end of the project lets you prove you have done it.

Answer 5

I'd highly recommend getting your hands on Enterprise Architect, which while not free, does offer academic pricing. It's an excellent tool for diagramming out projects both at a macro and micro level. It can also reverse engineer your source files into class diagrams, which would probably be a good start for you in documenting and reorganizing how your application goes together.

I'd also second @Klee's recommendations. Don't be put off by supposed "agile" tools, as they are really just best practice items that are also handy (if not mandatory) if you are following an agile process. But continuous integration (for example) is valuable no matter what your methodology. Moreover, unit tests (cppUnit?) are your friends. Unit tests should be very doable if you test the logic classes that are called by your UI, rather than testing the UI directly. There are also tools that could help you automate UI testing, but I'd try to get the back end stuff together first.

Good luck!

Answer 6

I believe that your problem has three dimensions

1. Programmer-Oriented
2. Program-Oriented
3. Program Maintenance - Oriented

Regarding the first problem, you might have programmers who don't understand the concept of what the program does (this often happens in corporate setups). But going by your question and the domain you are in, it seems that you and your team are in control of what the program does but just cannot translate it into a working program in quick time (To quote an example, I have heard about people having post doctoral degrees in Physics having trouble understanding pointers in programming and I am sure that Physics is much more difficult than C++). If that's the case, then your problem is mostly Program-centric (You must feel lucky enough that people around you can understand what your program is doing and can appreciate it).

In case of program-centric problems, one of my outrageous suggestions would be to move to a higher level of abstraction than C++ such as learning a new language as Python or Haskell (Python is quite easy to learn and if you are having good mathematicians, Haskell is just great). Moving to a higher level of abstraction would keep your code size down, easy to understand and maintain in the long run and effect changes swiftly. So you can have 10 lines of code in place of the original 50 lines of code. Also, having someone with expertise in computer programming would definitely help. Being in an university, you may also take the engage a few students in GUI and interfaces so that you could concentrate more on functionality. I also recommend the book Large-Scale C++ Software Design by John Lakos (Its quite a big book, you can become a proficient Python Programmer in the time you may took to read the book)

If you sort out of the first 2 dimensions of your problem, the third gets automatically solved. Since I believe you are working on a single large project, any decent project management software you get you through. Upfront planning is necessary. If you start coding straight away, you may get too much involved in the program that you would forget the big picture. Keeping things in mind wouldn't work for a big project. Put everything in paper (or in the computer). Use a wiki to share information. Regarding methodologies, I prefer the agile methodology (Simply, agile is a stylish name for incremental software development). I also suggest hiring someone with expertise in this area so that he takes care of these so that you could concentrate on your core program. The bottom-line here is that all project management methodologies are just a way to ensure the success of the program; so you can pick anyone that suits you and your team instead of selecting something that someone recommends the best. Also, the following software may also help you

• Free Mind - Mind mapping software
• Trac - Quick and easy software bug program and a wiki
• MoinMoin - Quick wiki to enable knowledge sharing about the program

Though the above tips would take some learning curve its worth it in the long run. And finally, programming is easier than Photonic Engineering (though not C++ Programming)

Answer 7

Like your question, any answer that is useful to you will be very long and vague - but the short answer is "Software Engineering"

I suggest you clear some free time, as much as possible if you are serious about a software development career, and start by visting Steve McConnells site. Programming is easy and can be taught in a semester, Software Engineering is an order of magnitude more complex and needs a lot longer to learn.