Should there be a formal role (internal or external) assigned to dev environments to control wasteful complexity?

Question

A few years ago, I worked for a small company that went so far into developing in-house frameworks that they dedicated their most senior developer, and their architect to developing a custom MVC framework and ORM from scratch. These two things were orthogonal to their core product. Unfortunately the support for this framework-driven approach came from the top, and it delayed delivery of revenue-generating software - and the frameworks produced were notably inferior to off-the-shelf alternatives, which compounded the delays. The company burned through cash quickly, and, eventually, everyone was laid off.

A later employer made a similar mistake - they got a highly technically skilled developer - a perfectionist, with a strong grounding in enterprise design patterns to grossly over-engineer a consulting solution for one of their clients, knowingly taking a loss, in the hope that the solution could be adapted and extended for other customers. The project overran significantly. But no other potential customers bit. A gold-plated strategic blunder that cost a small fortune.

In both cases there was a significant degree of overengineering. In both cases the company and project management were people with domain or analysis backgrounds, rather than a programming background. In both cases the software architects built overly-complex solutions to scratch their itches, and enhance their résumés, with some complicity from non-technical management. In both cases the architects had no stake in keeping costs down (apart from the risk of losing their jobs if the companies went bankrupt- which wasn't much of a risk, considering their high employability).

My experience suggests that it isn't an uncommon trap for dev shops to fall into.

Is there place in software projects for a formal internal or external technical adversarial role - a "Quantity Surveyor" - to use a building analogy- that can call out, or prevent wasteful over-engineering? Who is best suited to fill this role?

Answer 1

"Controlling complexity" is the job of:

1. The architect, whose job it is to ensure that the system and enterprise architecture is suitable for the current projects/products;

2. The developer, whose job it is to create designs and code that are not only correct but also maintainable, and also to review the code of other developers to ensure that they can understand it; and

3. The business/systems analyst or product owner, whose job it is to work out what the business actually needs right now, what can be finished later, or what might never materialize at all.

"Complexity" is an abstract concept anyway, and it could well be argued that "reducing complexity" is the very purpose of software in the first place, so it doesn't make sense to suggest a role dedicated to it - that's what the whole team should already be doing!

Assuming that a cursory look at the ROI and TCO demonstrate that the solution(s) is/are really and truly over-engineered, then I'm sorry to say that the architects and developers and analysts who were working on it simply weren't very good at their jobs. And the people in charge of that aspect are the managers or executives who hired them; possibly they've cultivated a culture of over-engineering and are partially at fault, or possibly they've just been getting bad advice from the implementation team.

I won't venture a guess at which one since I (probably) haven't worked for your company, but I'm sure you can figure it out for yourself pretty easily by just having a one-on-one conversation with one or a few of said managers.

---

Incidentally, developing in-house frameworks isn't always a bad thing. It's just that those frameworks should generally be based on observation and refinement of the current process. Frameworks or libraries that come about by refactoring existing code tend to last a long time. On the other hand, if people just dive in and start developing a framework with no context at all, then it usually becomes a liability, and quickly.

People have to be able to recognize the difference between conventions (most projects at an organization will use the same technology stack, with a certain amount of boilerplate code, but that doesn't mean jamming it all together into a "framework" is a good idea) vs. duplication (people are literally solving the same business or technical problems over and over again and the inconsistencies lead to convoluted designs and serious defects). Software businesses can and should recognize the latter scenario and actively take steps to, well, reduce the complexity. Frameworks do reduce complexity as long as they're not succumbing to the inner-platform effect.

Answer 2

Is there place in software projects for a formal internal or external technical adversarial role - a "Quantity Surveyor" - to use a building analogy- that can call out, or prevent wasteful over-engineering? Who is best suited to fill this role?

No, usually there isn't such position.

The trick is to hire right people for the job. People who are not going to over-engineer, and do just what is needed.

This may be done by using agile approaches (like implementing requirements in iterations, or using TDD or BDD). But again, if the architecture is over engineered, it is not going to help.