Is there really anything to gain with complex design?

Question

I've been working for a consulting firm for some time, with clients of various sizes, and I've seen web applications ranging in complexity from really simple:

• MVC
• Service Layer
• EF
• DB

To really complex:

• MVC
• UoW
• DI / IoC
• Repository
• Service
• UI Tests
• Unit Tests
• Integration Tests

But on both ends of the spectrum, the quality requirements are about the same. In simple projects, new devs / consultants can hop on, make changes, and contribute immediately, without having to wade through 6 layers of abstraction to understand what's going on, or risking misunderstanding some complex abstraction and costing down the line.

In all cases, there was never a need to actually make code swappable or reusable - and the tests were never actually maintained past the first iteration because requirements changed, it was too time-consuming, deadlines, business pressure, etc etc.

So if - in the end -

• testing and interfaces aren't used
• rapid development (read: cost-savings) is a priority
• the project's requirements will be changing a lot while in development

...would it be wrong to recommend a super-simple architecture, even to solve a complex problem, for an enterprise client? Is it complexity that defines enterprise solutions, or is it the reliability, # concurrent users, ease-of-maintenance, or all of the above?

I know this is a very vague question, and any answer wouldn't apply to all cases, but I'm interested in hearing from devs / consultants that have been in the business for a while and that have worked with these varying degrees of complexity, to hear if the cool-but-expensive abstractions are worth the overall cost, at least while the project is in development.

Answer 1

...would it be wrong to recommend a super-simple architecture, even to solve a complex problem, for an enterprise client?

It depends on the client. Many will be fine with a more straightforward approach. Some will think that you're incompetent because that simple approach will not solve their problems - the whole "that thing must be super cheap for a reason..." thing that causes some stuff to be way more expensive than it costs to make.

testing and interfaces aren't used

I would argue that there is no way in hell that you can cost effectively make non-trivial software without either of these. As a company, if you tried to sell me something without them, I would fear that you will saddle me with a completely inflexible, unmaintainable pile of steaming dog poo. (edit: to be clear I'm talking about interfaces here, not interfaces. Procedural and functional programs are not all piles of crap, even though they lack Java-style interfaces.)

if the cool-but-expensive abstractions are worth the overall cost

As program designers, we need to make educated guesses about where things will change, and draw lines there so that things that change can do so quickly, easily, and correctly. Good abstractions will save you money over time because inevitable change becomes cheap.

But make no mistake, the term "architecture astronaut" exists for a reason. You are not always building the space shuttle. Adding bad abstractions that either exist in the middle of things that need to change, or exist to support change that does not exist... they can cost you.

But in my experience, I've heard people complain about over abstracted code, but I've only seen projects fail due to under-abstracted code.

Answer 2

To Architect, Or Not To Architect

To rephrase Einstein, "things should be as simple as possible - but no simpler." It's important to understand the true use of some architectures - and they aren't necessarily for individual performance, and often not for job satisfaction either.

Sometimes architectures are so complicated because an enterprise project is just so large, with so many people at work, that the architecture is largely to allow so many people to work towards something without everyone tripping over everyone else. They are, at times, little more than complexity added that reduces individual productivity and makes work harder - but permits large numbers of people to work. This works when the added effort outweighs the individually decreased will to live, err, I mean "productivity".

Is this your client's situation? Well, if their budget isn't in the millions for this project, maybe not. If it's under a 100k, then almost certainly not. You build a concrete bunker for a nuclear reactor, not a wood burning stove :)

It is not necessarily wrong to suggest "the simplest thing that could possibly work" for your client, if that's what the situation calls for. It might be the best possible way to handle it. The easiest way to see is compare it to commercially available/open source projects that are designed to accomplish a similar task (this is your due diligence market research, anyway), and see what it took them to do what they did.

Of course, when you are working with millions of dollars you should be aware of some perverse incentives in play. In short, "ass covering": if you recommend a super complex architecture system that requires a PhD to understand properly, when it goes to hell who's fault was it? Not yours, of course - you suggested only the finest in sophisticated technologies that everyone says is the best - but they are "complicated" and "difficult", so the blame is with the people who tried to implement it. They must have just not been smart enough or skilled enough...in other words, lots of advice is given to ensure you don't look bad when projects fail, which they inevitably must as not all projects are successful. It's bad juju, but it is a fact of our reality.

Failure To Follow Through

I would suggest that failure to continually update tests and follow-through on architecture decisions (tossing plans when they change instead of updating them, etc) is why so many systems end up being thrown out and rewritten from scratch or wholesale replaced with a new 'solution' a few years down the road.

When things that enforce clarity and consistency are thrown out for expedience, it is easy to sometimes think you were very sly and are really "Getting Things Done" when in fact you made future changes harder and more complicated; it is very common to steal from future productivity for immediate gains. Its bad specifically when no one realizes that's what's happening.

Testing, for instance, is specifically best suited to when things change. You change the tests to reflect what the new direction is, and then go about fixing all the stuff that is now "broken" because people changed their minds or came to a realization that they missed something last time. This is like giving up sleeping because you have a lot to get done this week - its going to make things harder.

In Conclusion, "It Depends" (Helpful, Isn't It?)

Architectural complexity is an overhead cost, like renting a larger space for a brick-n-mortar business; it only makes sense when there is real complexity that needs to be managed. If it's a team of 3 working on a solution that is relatively conventional, simple MVC and minimal tooling is probably entirely sufficient.

If it's a complex, multi-subsystem, enterprise software solution, with millions on budget and at least a half-dozen teams of 3-5? You're going to be begging for the sweet, sweet love that only round-trip engineering on object oriented program analysis and all its charts and diagrams and models can give. Anything to help you keep the dozens or hundreds of sets of eyes from unblinkingly staring at you, wondering what they are supposed to do today and how they are supposed to get anything done at all.

Answer 3

Welcome to consulting.

A great place to make a ton of money quickly delivering features to clients.

If you seek to work creating maintainable, adaptable, beautiful software that works well both now and in the future I recommend you work in a specific industry such as healthcare or government where you can carve out a better niche for quality work.

Answer 4

What you gain from a solid design, is a more reliable change cycle over the long term. The cost is initially higher because you need properly trained programmers and a proper design, however the intent is to keep the cost of later changes from skyrocketing out of control.

Without a solid architecture changes are only cheaper at the beginning. As the project evolves, changes become more and more costly. This, combined with a desire for cheaper labor and faster iteration, could kill the project as changes will eventually cost more than the company is willing to pay and take more time than the company can wait for. Later changes that could have taken an average team and average time will instead require a superhuman team and a lengthy time to sift through the years of code and refactor enough to safely make changes.

If the client understands and wants to make this sacrifice, then make whatever changes are needed by whatever method is fastest. But it is unwise (and possibly dishonest) to make those tradeoffs for them, as this process will affect their business decisions.

---

Part of the job is to understand all those abstractions you listed, not just to ask "Are these necessary?" but to understand when and why they are necessary, and to figure out how to apply those patterns as part of working with the software.

Another part of the job is to understand that clients often judge work by what they can see (the ui, the functionality), and they should be educated about what they can't see (the state of the codebase).

• See this talk regarding the cost of poor design over time
• See this talk regarding stable productivity
• See this SO answer regarding the value of unit tests

Answer 5

Complicated is relative. When I first started learning to program the concept of pointers especially linked lists was complicated ("How can you have something that has itself as a field"), but once I got my mind around the concept, that was simple.

When I started learning functional programming the concept of a lambda seemed complex. But then I related it to the idea of a Function pointer from C++ and it was simple. And so on with currying, and completions, and so forth and so on.

When I first started learning TDD, I thought it was complex or that it added overhead to the development process. Now I don't feel my code is complete unless I have tests at least covering the happy path. Dependency injection and using interfaces instead of concrete classes seem complex at first. But they allow you to build your code with the assumption that you'll work out the details of the dependencies and focus on the higher level constructs first.

Check out Clean Code and The Clean Coder by "Uncle Bob" Martin. The first talks about how to code better. The second talks about how to be a better coder. There's a difference between the two.

To give an analogy, I'm learning guitar. I'm at the stage where I can play simple melodies and chords. Transitioning between chords and positions is difficult. By practicing more, I'll be able to hit those transitions easier and it'll become second nature to me.

I use the repository/unit of work pattern so that I can use an in-memory repository while I'm building the basic functionality and convert to a "sql lite" or "odata" repository as I get closer to delivery. Note that this allows me to complete more functionality even before the final infrastructure is in place (or even decided on). The abstractions allow me to create mocks and stubs more easily so that I can verify the code that depends on the abstractions works as expected. Again, I can complete features from the top down.

I would say it's wrong to create the infrastructure first. The simple fact is that it is anti-agile. If I spend a sprint on "infrastructure", I'm building stuff that I'm not even certain I will need in the end and the users will have nothing to look at to provide feedback.

That is what the abstractions provide. The more you practice using them, the more they become second nature, and you'll look back at this discussion and marvel how far you've developed. Just like hopefully, I'll wonder in a few years how chord transitions were ever a challenge for me.

Answer 6

I do not fully agree with your statement that abstraction automatically increases complexity. Abstraction is the primary reason why one nowadays doesn't have to resolve to machine code.

The problem sometimes with abstractions is that they tend to focus on the inner workings rather than hiding the complexity to its end users. Abstractions should isolate logic and promote reusability. Furthermore, I think testing is a great way to challenge the architectural design. Mainly because it forces you to look at your code from a user point of view.

The problems you mentioned about lack of time, is the primary cause for bad design. Good design usually doesn't come instantly, but through iterations and refinement. How we usually extract requirements from our customer is via user stories. This way the customer can think about what and how he wants to use the system. Furthermore, if the architect is aware of the intentions of the system then appropriate design decisions or abstraction techniques can be chosen to add the flexibility to the system. Not because they are cool or nice.

Answer 7

Addressing your questions...

...would it be wrong to recommend a super-simple architecture, even to solve a complex problem, for an enterprise client?

Absolutely not.

From the Client Perspective

As stated above, it largely depends on the client. It also depends on your ability to accurately gauge which solutions are right for your client. While there is always going to be a perceived cost to desired value, as a consultant it is your job to set the appropriate expectations of the client. In some cases, you'll have to meet that perception. In others, it will be in your best interest to correct them. After all, you should be the expert for your client. And if you aren't, you should have the knowledge to be able to become that expert. That's what they pay you for.

From the Developer Perspective

The hardest part about choosing which architecture to use is, often, correctly estimating the amount of work required to utilize the technology to meet the specific needs. This can very quickly lead to projects that fail client expectations. Understand that some projects are actually made faster using these "complex" pieces of code that you mention. It is also understood that some are not. You have to provide that measurement, based on what you or your team knows.

Is it complexity that defines enterprise solutions, or is it the reliability, # concurrent users, ease-of-maintenance, or all of the above?

While specifics may vary, in general, an enterprise solution is a software solution that applies to a wide co-mingled audience. The number of concurrent users may or may not be a factor, though often is. The total number of users, frequently in a variety of business roles, is one of the largest determining factors as to whether the solution is "enterprise".

Many enterprise solutions are highly complex, but some are quite simple. While enterprise gives an air of reliability (and should certainly be held to a certain standard), different solutions have different levels of reliability.

Ease of Maintenance is something that I think every developer (independent or team member) strives for, it is not necessarily as easily achieved. What is important is that there is a maintenance procedure that has firm guidelines, rather than it be "easy". Remember, different code-bases will have substantially differing levels of ease depending upon the philosophies, methodologies, (business) environment activity, and complexity of the code.

Reacting to your other statements...

In all cases, there was never a need to actually make code swappable or reusable

There is often never a specific need to do so. This should be your goal, at all times, however. Consider this... You might have a client that requires the ability to access or display their calender from the web page. If you make your own code reusable, then when another client asks for the same thing, you already have some of the work done. If you don't, then you have to do it all over again. Every client problem is often one that another client in the future may need. In other words, every client you work with should have the potential to reduce the cost of work for your future clients (not necessarily the cost of the product).

and the tests were never actually maintained past the first iteration because requirements changed, it was too time-consuming,

I would argue here that the testing methodology was not abstracted enough. I recently used a piece of code that did its own unit tests directly within itself. A custom assert and expect function were made that accommodated the needs of the project. Anytime a unit test was needed, it could be applied without even adjusting the code. In fact, the code is actively distributed with the asserts and expects still in there. It made those checks as part of the working code.

... deadlines, business pressure, etc etc....

I have often found that extra business pressure and deadlines impeding the coding process have been the fault of the developer, not of the client. While this isn't always the case, many times business pressure is caused by a perception of failure to meet client expectations. When deadlines impede code, it is often because the developer failed to accurately gauge the amount of work required for usable functional code. In other words, schedule them (clients expect it), measure them (future clients expect it), perform them (users require it), and get paid for them (your contract should demand it).

Answer 8

A lot of what this question asks is incredibly subjective, I think it's important to bring up some data on the compounding costs of architectural complexity. There is a really interesting case study coming from MIT Sloan which measures

• decrease in developer productivity
• increased turnover
• and increased defect density that results from more complexity.

Basically, I would say there's some happy medium which every codebase should strive to strike between (A) simplicity so that developer productivity is high and (B) foresight so that functionality isn't compromised when new features are being added.

The case study was on this one company with a monolithic codebase that had a high-complexity core (think like utilities and high abstraction stuff) and lower-complexity periphery (think newer features with few dependencies). Between the periphery and core, developer productivity decreased by 50% (crazy stuff) and defects increased 2.6x.

Answer 9

I can relate to your doubts regarding complex software development techniques, and would guess that there are a lot of people that can, for exactly the reasons you named: They add an overhead to the initial creation of a new system while not always instantly showing their benefits.

That does clearly not mean they do not exist, for every concept there is a reason, most of the times a good one. In total they should reduce the work needed to finalize a project and enhance it afterwards.

In theory, of course, and I would bet you learned all those theoretical benefits and now more than ever you see them fail. And that may have a good reason, too, using a concept right is not as easy as understanding why it is used, and while you may have the experience it takes to decide when to abstract and when not, your clients may not yet.

Using a concept at every possible point in your project can cost you a lot of time (see Telatsyns answer for a neat simple sentence on that) and even using a concept at the right place can cost you, if you are not experienced enough using it. As you said, those are no simple concepts but sometimes really complex ones that need to be fitted to your situation, a theoretical understanding can not be enough to use it fast and right.

And for those reasons it is of no surprise when your clients without really noticing drift away from the concepts and abandon their tests and abstractions and start working simple again.

Regarding the big barrier that software engineering can be, I suggest this article to you: The Seven Stages of Expertise in Software Engineering I stumbled over it somewhere on this page and it expresses some really interesting thoughts and experiences.

Answer 10

I would say that if you are developing applications for the enterprise then you or someone in charge of your projects (Proj mgr) should have a better feel for the amount of work that should go in to the projects. If you are seeing a waste of time then you should discuss your view with the Proj mgr. The way I feel and operate is always on a per basis type of judgement, but I always look out for that client that will throw you a curve ball. I would error to doing a good job by dotting your I's and crossing your t's. I would also make sure that I have the proper application lifecycle management in place so that most of the overhead is automated so that it ill save time etc... I hope that helps!