8

A few years ago, I worked for a small company that went so far into developing in-house frameworks that they dedicated their most senior developer, and their architect to developing a custom MVC framework and ORM from scratch. These two things were orthogonal to their core product. Unfortunately the support for this framework-driven approach came from the top, and it delayed delivery of revenue-generating software - and the frameworks produced were notably inferior to off-the-shelf alternatives, which compounded the delays. The company burned through cash quickly, and, eventually, everyone was laid off.

A later employer made a similar mistake - they got a highly technically skilled developer - a perfectionist, with a strong grounding in enterprise design patterns to grossly over-engineer a consulting solution for one of their clients, knowingly taking a loss, in the hope that the solution could be adapted and extended for other customers. The project overran significantly. But no other potential customers bit. A gold-plated strategic blunder that cost a small fortune.

In both cases there was a significant degree of overengineering. In both cases the company and project management were people with domain or analysis backgrounds, rather than a programming background. In both cases the software architects built overly-complex solutions to scratch their itches, and enhance their résumés, with some complicity from non-technical management. In both cases the architects had no stake in keeping costs down (apart from the risk of losing their jobs if the companies went bankrupt- which wasn't much of a risk, considering their high employability).

My experience suggests that it isn't an uncommon trap for dev shops to fall into.

Is there place in software projects for a formal internal or external technical adversarial role - a "Quantity Surveyor" - to use a building analogy- that can call out, or prevent wasteful over-engineering? Who is best suited to fill this role?

  • 3
    I'm not sure you could justify a position for someone who could be replaced by a piece of paper with "YAGNI" printed on it. ;) – vaughandroid Oct 11 '13 at 16:02
  • 2
    In a draft, I described the role as a "YAGNI enforcer". Unfortunately awareness of YAGNI doesn't mean it will be observed, especially in non-agile environments. – user104662 Oct 11 '13 at 16:43
  • 3
    "a perfectionist, with a strong grounding in enterprise design patterns to grossly over-engineer" != "great developer" – Evicatos Oct 11 '13 at 19:27
  • 1
    It seems maybe an investor or significant stockholder with a few brain cells would be a good candidate for this. somebody focused on money, spending the money, how soon are we gonna get back the money. not that investors aren't smart, i think sometimes they don't care either because it isn't their money usually. – Andyz Smith Oct 11 '13 at 21:47
  • 2
    Sounds like the most senior developers (and architect if you have one) overseeing the projects should be the very ones monitoring this sort of stuff. It also sounds like a lack of a real project manager in both cases to motivate (annoy) developers to deliver. – Rig Oct 12 '13 at 14:04
6

"Controlling complexity" is the job of:

  1. The architect, whose job it is to ensure that the system and enterprise architecture is suitable for the current projects/products;

  2. The developer, whose job it is to create designs and code that are not only correct but also maintainable, and also to review the code of other developers to ensure that they can understand it; and

  3. The business/systems analyst or product owner, whose job it is to work out what the business actually needs right now, what can be finished later, or what might never materialize at all.

"Complexity" is an abstract concept anyway, and it could well be argued that "reducing complexity" is the very purpose of software in the first place, so it doesn't make sense to suggest a role dedicated to it - that's what the whole team should already be doing!

Assuming that a cursory look at the ROI and TCO demonstrate that the solution(s) is/are really and truly over-engineered, then I'm sorry to say that the architects and developers and analysts who were working on it simply weren't very good at their jobs. And the people in charge of that aspect are the managers or executives who hired them; possibly they've cultivated a culture of over-engineering and are partially at fault, or possibly they've just been getting bad advice from the implementation team.

I won't venture a guess at which one since I (probably) haven't worked for your company, but I'm sure you can figure it out for yourself pretty easily by just having a one-on-one conversation with one or a few of said managers.


Incidentally, developing in-house frameworks isn't always a bad thing. It's just that those frameworks should generally be based on observation and refinement of the current process. Frameworks or libraries that come about by refactoring existing code tend to last a long time. On the other hand, if people just dive in and start developing a framework with no context at all, then it usually becomes a liability, and quickly.

People have to be able to recognize the difference between conventions (most projects at an organization will use the same technology stack, with a certain amount of boilerplate code, but that doesn't mean jamming it all together into a "framework" is a good idea) vs. duplication (people are literally solving the same business or technical problems over and over again and the inconsistencies lead to convoluted designs and serious defects). Software businesses can and should recognize the latter scenario and actively take steps to, well, reduce the complexity. Frameworks do reduce complexity as long as they're not succumbing to the inner-platform effect.

  • Good answer. I'd like to add bullet point 4.: you will need a management which allows the other roles (1-3) to pay back technical debt. This will speed up development in the current releases because 1-3 know they can refactor later as needed. If management always uses up all available ressources for new features, architects and developers are pushed towards inventing flexible frameworks which try to solve all potential future problems. That is nearly impossible and creates the problems described by the original author. – Andreas Huppert Oct 13 '13 at 12:41
  • 1
    @AndreasHuppert: I certainly agree that management needs to understand the idea of refactoring and that teams can't have 100% of their time committed to new features - for well-run teams, that figure should be more like 20% or even lower, the rest being occupied by planning, testing, code reviews, automation, refactoring, debugging, etc., and of course the usual overhead and interruptions. But I'm not sure if I agree that a lack of time is what leads to these horrible inner platforms. If a team doesn't even have time to refactor, where could it possibly find the time for those? – Aaronaught Oct 13 '13 at 14:39
  • 1
    Because it is possible to sell investment into a platform/framework as investment with a good ROI to management. Refactoring has much less glamour and is hard to sell. – Andreas Huppert Oct 13 '13 at 22:01
5

Is there place in software projects for a formal internal or external technical adversarial role - a "Quantity Surveyor" - to use a building analogy- that can call out, or prevent wasteful over-engineering? Who is best suited to fill this role?

No, usually there isn't such position.

The trick is to hire right people for the job. People who are not going to over-engineer, and do just what is needed.

This may be done by using agile approaches (like implementing requirements in iterations, or using TDD or BDD). But again, if the architecture is over engineered, it is not going to help.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.