TL;DR
Your experience is typical. As noted in other answers the Agile Manifesto, eXtreme Programming (XP), Scrum, Kanban, Lean and many other techniques are popular to avoid the inefficiencies you point out.
Your question is slightly controversial is it asks "why is a SDLC so inefficient". Yet not all SDLCs are this inefficient. You can have a lightweight SDLC when using, say, Kanban and do continuous deployments into production multiple times each day.
A few very good books for developers that discuss the problems you have identified and possible solutions are:
- "The Phoenix Project" by Gene Kim and Kevin Behr
- "Agile IT Organization Design" by Sriram Narayan
- "Ship it!" by Jared Richardson and William Gwaltney.
I would suggest that you find a job with a company that takes a different approach than your current one to broaden your horizons and improve your job satisfaction.
Why Are Developers Treated As "Mushrooms"?
There is an old joke:
I am treated like a mushroom. I am kept in the dark and when the light rarely comes on for a minute I am buried under another ton of manure.
If you work in a large bureaucracy as you describe it is very frustrating and unrewarding to be a developer. Unfortunately, the problems you describe result from company structure, the annual financial budgeting process, and how the bureaucracy functions. In large companies, this creates many hand-offs to pass requirements from the Sales or Product team back to the developers. As you point out this causes long delays in the feedback loop. Efficiency, quality and features all suffer when there are a long feedback loop and multiple hand-offs.
Having a decision-maker such as a Product or Sales team member spend even as little as half an hour a day talking to developers would be vastly more efficient. They could answer ad-hoc questions and feedback on partial work. To maximise the feedback loop the features should be broken down into small units. A very common approach is to work in two-week sprints. This reduces the window where they can be any misunderstanding on how a feature should work to be less than two weeks. A very common approach is to have a retrospective after every two weeks. If there was confusion about a feature in the previous sprint the retrospective can be used to change the way the team works or documents requirements to avoid the same wasted efforts occurring in future sprints. This could be as simple as "next time we attempt something like X again we will all sit in a room for three hours and sketch it on the whiteboard first". Flexibility and communication are far more valuable than documentation and processes when converting concepts into working software.
The forces that push against creating a single empowered team are driven by corporate politics. The people in the 'hand-off' chain between the developers and stakeholders such as Sales/Product would likely feel very uncomfortable to be bypassed. If your job is to "deal with the development" so that "Sales/Product can focus time on customers" why would you step back and let developers just work directly with the Sales or Product people? The very existence of your job, set up by more senior managers, implicitly states that the company believes that developers should not speak directly to Sales or Product. It is therefore clearly the responsibility of the intermediaries to prevent direct communication. This perversely leads to a very inefficient development process which is harmful to the company and that allows more nimble competitors to beat them.
Understanding why the situation above comes about and how to change it is a large topic. It spans concepts like accounting, corporate structures, employee incentives and many other MBA topics. The book "Agile IT Organization Design" by Sriram Narayan does an excellent job of explaining why these systemic problems occur and how to fix them. The book argues that companies fail to work efficiently due to applying manufacturing management techniques to software development.
Fundamentally software development isn't actually like manufacturing at all. Developers should not be treated like blue-collar workers working on a factory line. Software Engineering is inherently entirely a design process from end-to-end. The act of writing code isn't an act of transcribing a local design into machine instructions. It is more like the act of the developer creating the final business solution within their IDE while seeking inspiration from the written requirements. Developers don't actually "implement features" when they write code. Instead, they interpret the requirements and design the concrete solution as they write code. Applying manufacturing management techniques to this process is extremely counter-productive. It stifles the productivity, creativity and ingenuity of developers and ultimately causes them to quit software development.
To be clear I am not advocating "developer anarchy" as the ideal way to develop software. Teams should use a lightweight governance process and SDLC. I am advocating for small, empowered, mixed discipline teams to rapidly iterate on features and fixes with zero intermediation between them and their key stakeholders.
Why Does Shipping Software Require Delays And Handoffs?
There is an industry concept of DevOps as a culture where developers are empowered to do continuous deployment into production using rigorous automation. Famously highly productive companies release software many times a day. There is a direct correlation between how frequently software is deployed and how successful a company is. For example:
The range of software deployments went from 1,460 deploys per year
(calculated as four deploys per day x 365 days) for the highest
performers to seven deploys per year for low performers.
As mentioned in the previous section the problem is that software delivery is treated like manufacturing. If you are making a car, and it has a safety design flaw, it can bankrupt your company when you need to recall and fix all the bad cars. At the end of the last century enterprise software was shipped on a physical CD. A factory would print thousands of copies of the CDs that would be physically distributed to suppliers. Shipping a service pack was also done on thousands of CDs. Treating software development as you would treat manufacturing seemed like a very good idea.
It is typical in manufacturing to batch up work to make it efficient. Batching features into monthly or quarterly release appears to optimise the requirements analysis phase (let us not distract Sales/Product too much!). Batching up testing work into two or three weekly drops appears to optimise the testing phase. Deploying a large release mostly by hand using a Tech Ops team seems to be easier than investing in building a fully automated deployment pipeline. Running and monitoring monoliths appears to be easier and more efficient than running lots of microservices. Yet these are all local optimisations. These local optimisations make the end-to-end process of delivering new software poor when compared to high performing companies.
The forces that push against empowering a developer team to do rapid deployments many times a day are driven by both software architecture and corporate politics. If your job is to do "system testing" protect the business from bugs or to "do deployments" to keep developers away from the live environment why would you step back and let developers have a continuous deployment pipeline? The very existence of your job, set up by more senior managers, implicitly states that the company believes that developers should not be empowered to deploy code rapidly into any of Test, UAT or Live.
Then there is the question of whether technically you can continuously deploy small features or bug fixes into the live environment. If you have a monolithic software architecture this is hard. You get all the inefficiencies that you point out. Using feature toggles can help. Better yet a deployment system that lets you do canary releases or blue-green deployments.
One of the main drivers to the adoption of a microservices architecture is so that teams can continuously deploy small features or bug fixes. This makes the company far more productive at the expense of the complexity of running a large number of microservices rather than a simple monolith. If you have a microservices architecture then a Service Mesh solution like Istio will let you tag only a small percentage of traffic to use a new version of a microservice deep within the system. This is known as Traffic Shifting. Swap in a new version of your microservice to a small population of customers, monitor for errors, then gradually roll it out. If unexpected problems occur then shift the traffic back to the old version. Else if the problem is only for one or two customers you can deploy a quick fix just for them and shift only their traffic onto that quick-fix version. That then gives you time to go back and fix the root cause to be able to run a single version of the code for all customers. This may sound like magic but it is possible. Companies that can do this are wiping the floor with competitors in terms of both software costs, quality and time-to-market.
Why Release Little-And-Very-Often?
I have direct experience of working at a large global financial services company that mandated that software teams doubled the number of releases, and halved the number of standard defects, and quartered the number of critical defects, every year. Senior management directly tracked that this was happening across a few thousand people working across a few dozen large scale software systems and over a hundred small systems. Some big teams needed to go from quarterly releases to monthly releases, then fortnightly, then weekly, then mid-week. Some small teams were already doing fortnightly releases at the start so had two years to move to mid-week deployments. This wasn't seen as an optional efficiency drive. It was seen as something necessary to allow the company to survive in the face of stiff competition from more nimble competitors.
Systems that couldn't make the transitions due to architectural issues were then flagged as "legacy" and were no longer investing in.
The strangular pattern as used to start the build of replacement systems built as microservices. These new systems aim to ship code into production continuously. Why make this investment? Surely the business could just give the legacy monoliths "a pass" to keep on doing quarterly or monthly updates. Remarkably while releases were doubled across many of the systems their defect rates also did half and critical outages did go down to a quarter in the first few years. So not only was it now very obvious that it was hard to introduce new features into the monolithic systems it was obvious that they were very poor performers. They systematically caused more pain to the team maintaining them and gave a poor customer experience. This was in part due to very long lead times in fixing issues leading to a patchwork of expensive manual workarounds.
It may seem counter-intuitive that you can deploy into production every day and get a higher quality outcome than having a long test cycle. Yet it is the natural result of the more automated testing, shorter feedback loops, more focused and motivated developers, improved software architecture, and improved automation required to do continuous deployments. For example, developers were introducing much smaller changes and rolling any problems back much faster using feature toggles. It is often quoted that research says that a bug that gets into production over a hundred times more expensive than not caught earlier in the SDLC. The answer isn't to run an inefficient SDLC that makes your company uncompetitive such that more nimble competitors put you out of business. The answer is to do everything possible to reduce both scope and duration of bugs in production while making a more efficient SDLC that can produce higher-quality software in much smaller units that can be continuously improved upon.
At the same time as achieving these successes, job satisfaction increased across developers. Usually, the organisational restructuring needed to achieve such changes leads to some of the best people leaving a company. If it actually leads to people feeling empowered and more productive it helps with both hiring and retaining talented staff.
