When systems get larger and larger, how do you keep a global understanding of your system? [duplicate]

Question

Today I realized painfully that for some decisions you need a good overall understanding of the system. Otherwise, the risk is too high that your assumption turn out to be wrong.

Imagine that you are a developer for an online web shop. To understand the system, you have to understand many connected subsystems, for example:

1. How to receive and process product information from various suppliers
2. How the customer can search and order products on your web shop
3. How the orders are processed and managed by your customer service
4. How your SAP system handles the invoicing process
5. ...

The larger the system becomes, the more you have to understand.

If that knowledge is lacking, sub-optimal solutions where developed when specialized teams worked together. Specialized teams, which only understood their part of the system in detail.

To deal with that problem, our company changed the strategy, so that one development team always has to be responsible for all aspects of a feature. Even if it involves the complete process chain. (It is kind of a feature team, but not from teams working on different subsystems.)

What are effective strategies for developers to keep their system and operational process knowledge up to date?

I think good system documentation is a key, but I'm afraid that there is a point where the human mind cannot scale as fast as the system involves. At some point to have to simply, but that simplified assumptions can turn out to be costly mistakes. When you have to implement and maintain the code, you just have to know the exact details.

As a developer, I currently have to face a difficult conflict of interests:

1. I need to spend more time to understand our system and operational process.
2. I need to develop and maintain our code.

As time is limited, 2) mostly wins. The result is that I mostly gain deeper knowledge along the way, and some half knowledge from casual conversations.

Do you know how huge companies like Amazon solve that problem? I would assume that no single human is capable of understanding such a complex process, and be able to contribute code at multiple subsystems at the same time.

Answer 1

Understand the components

Let's start with a fundamental assertion that, I hope, is already self-evident:

Smaller teams are more efficient, creative and productive than larger ones.

And a simple, but fundamentally important maxim:

To cook an elephant, break it into smaller pieces.

In programming terms, we do this by encapsulating, abstracting and modularizing, always striving to create coherent, easily understandable, functional and consistent interfaces between components. Each part of the elephant gets its own conceptual black box; once the internals of each black box are designed, coded, tested and accepted, nobody has to think about the box's insides anymore, unless a new requirement dictates a functional change.

This works at the people level as well. Because smaller teams are more efficient than larger teams, larger companies (especially matrix organizations) frequently have several smaller workgroups, each of which is responsible for a certain part of the software.

Document the interactions

As you've probably already figured out, the problem is not the black boxes, it is understanding the interactions between the black boxes. There are a number of techniques for dealing with this, all of which involve more time and resources:

1. Class documentation
2. Integration tests
3. Architectural diagrams
4. Data flow diagrams

And so forth. Of course, you have to have people who can read and understand this documentation and apply it at a high level. It must be available to the software developers, and kept up to date.

We're dreaming, right?

Where I work, we run a pretty lean shop. But we do our best to create these artifacts. They are, in fact, required elements of the overall software product.

Divide and conquer

Basically, what you want is for your small teams (and your software baby elephant) to look more or less like this:

And your overall software team (and grown software elephant) to look something like this:

The real challenge is this: how do you maintain your information flow at a high level of quality and productivity? Within systems design, the way that is done is by carefully designing the communications between nodes, so that the interfaces, actors, objects and messages are well-defined, consistent, clear and understandable.

Remember, we're talking about black boxes here. We already understand the black boxes; the key is to understand the interactions that take place between the black boxes. A sensible architecture can mean the difference between well-organized and easily understandable software, and a big ball of mud.

And finally...

Be clear about what you want

If you have unclear or underspecified software requirements, it's not realistic to have a good understanding of the software. Requirements should be understandable in a way that the stakeholder can clearly understand when a feature is fully completed, and there are specific, identifiable and measurable metrics by which success can be declared.