Good, simple reasons for having multiple environments

Question

Throughout my career I had worked at companies that had a collection of different environments for different purposes. We always had more or less our desktop environment, a test environment, a QA environment, a staging environment and a production environment. This went for both servers/applications and any data sources we were using.

When I started at my current company I found that 90% of the apps were either developed on a desktop environment against production data sources or developed directly on the production server depending on the platform. This wasn't particularly surprising, as I was hired in part to make changes to improve the way the development team functioned, which was clear from my interview process. We slowly started to turn the philosophy and pretty soon, most of the apps could be run in either a desktop, test or production environment. Not too long after that staging came around as well.

Now most of our developers see the benefit of this methodology and defend it vigilantly. However, we have a number of legacy apps that never got migrated. We also have a number of legacy programmers who think of this as a waste of time. Unfortunately, we got lip service but never full buy-in from management. We got what we thought was a commitment to invest substantially in this about a year ago, but nothing materialized despite the considerable planning that we put into it. Now we are finding that we need more and more environments. We need help from the server/network administration teams for setup and we need participation from the business stakeholders to support the release cycle. We are at a place now where a project can function what reasonable developers would consider "normally" only if you have the right people on the project and the time to set up the proper environments.

I'd love to present a complete argument, but management really has no time and interest in hearing me out until there is a critical issue. I can't really articulate the benefits simply as it always just seemed second nature to me. I was wondering if there are any good, simple, irrefutable reasons for the separation of environments that would get managers lacking development experience to support this idea?. Are there any good resources/literature on the topic?

Answer 1

The answer: Money

I don't care what the actual reason is. Money MUST be at the root of all of your reasoning, especially when dealing with management.

If we both sat in a room for 2 hours, we could come up with dozens of reasons why it is better to have multiple environments.

Here's the problem: If the reasons are not based on money, then none of them matter.

Programmers are not hired to be smart. They're not hired to be creative. They're hired to increase revenue -- either by earning money or saving money. If you're not doing either one of those, you'd better get your resume together.

When looking at it from that standpoint, the answer is simple:

Having only one environment increases our downtime and results in lost revenue. Multiple environments allows us to protect our profits by giving our users a front-end that is just as reliable and dependable as our company.

Repeat it every day.

---

There are some great comments below that add some real value to this answer, so I'll mention them:

• Karl Bielefeldt had a great point when he mentioned that Cost/Benefit analysis is an important factor. An economist might refer to it as the opportunity cost of pursuing multiple environments. While it may be surprising to hear, there are scenarios where multiple environments may not be the answer! If the website of your company is a very minor addition, then unexpected downtime may actually be the more cost effective way of doing business. This doesn't sound like the position you are in, but it is worth mentioning.

• BlairHippo had a good point in that you should feel free to make it seem like a catastrophe (and if you lose your data, it is!). Liability is a great tool for persuading managers, but still for the same reason--lawsuits are expensive. Avoiding them saves money.

---

As an addendum, I found this article to be quite good. It doesn't directly answer your question, but enables you to recognize how programmers are viewed to management, which in turn, leads to this answer. Good read.

Answer 2

Single Point of Failure

By not having a development or staging environment you have a Single Point of Failure for those legacy applications. Management will hear you if you describe the legacy applications in those terms.

You need to be able to pitch your message in sound bytes that makes sense to them. Take the "Programmer Speak" out of the discussion and replace it with "Manager Speak". Also pretend you have a 30 second elevator ride to get your point accross.

I had a situation where my boss was an Infantry Marine. I kept telling him I needed software tools and computer training for my Marines in order to be more productive. He didn't get it. I finally went into his office one day and told him things were f'd up.

I said something to the effect... "If I was fighting a war I'd be using sticks, rocks and tree branches. What I need are grenades, bazookas and machine guns." He got the message.

Answer 3

Is it actually critical?

I can understand the desire to use separate environments. The non-obvious question is:

Is it actually critical to migrate a legacy system?

I think most technically minded people tend to focus on the academic question of which way is better which is fine in academia. In business though best doesn't always win out. I'm not saying this to be negative, or start a flame war. I am stating the obvious, or what should be obvious to those of us who have been in the software business for a few years.

All business decisions are typically made based on the perceived cost/benefit. So the question the business is probably asking is:

Is the cost of the additional system and development investment in a legacy application worth the benefit versus putting that same investment into another project/product?

I have and still do regular cost benefit analysis to make determinations not just in migrating/rewriting software, but in day-to-day decisions a lead typically engages in. I've passed on rewriting/migrating old software because it had limited life and therefore value.

Separating Environments

The business reasons to separate environments.

• Less risk in releases, and bug fixes. Prove it with numbers. How many times has the product failed and cost customer revenue because of a bad release/bug.
• Less risk in development. Accidentally blowing away the dev db is different from accidentally blowing away the production db
• The ability to clearly separate roles and access ie. better security. limiting the number of fingers in the production pie is a good thing
• The ability to separate environments, and the practices and procedure that go along with this style of development allow for future build out into Cloud Systems.
• The separation of environment should force efficiencies in replicating environments which may be useful in scheduled as well as dynamic scaling.

Answer 4

It sounds like you have all the "right" arguments already in place. Instead, you are experiencing a "red-herring", if you will. Or, "chasing the carrot"

management really has no time and interest in hearing me out until there is a critical issue

That's what I consider to be the real problem. In my experience, if a company has sub-par development practices as poor as you describe. It's not simply a matter of "we didn't know any better". Rather, its a compilation of technical debt caused by an upper management team that doesn't know (care?) about the problems it presents.

In such cases, a good pep-talk is not going to suddenly swing things your direction. Maybe a severe trauma (product failure visible to client and directly tied to poor practices), but I'm sure sane savvy techies before you have tried the talk thing.

My suggestion is to either suck it up and take things for what they are or look for a new position.

Answer 5

How many groups of people plan to be working on the app at a time? Usuaslly I've seen one environment for each group of people. This is the Developers (they get a DEV environment and a DEV Integration environment - some would say not 100% necessary, I'd say it varies by project), two testing environments (one group of testers doing very detailed testing, the other for high-level QA testers - usually they are actual business users, not trained testers). There is also usually an isolated Performance testing environment (so you can test huge volumes of data, simulate huge volumes of users, etc... g).

Why all these environments? So different groups can test different features without stepping on each others' toes. If developers and testers work in the same environment, it's a nightmare: a tester can open a defect on a feature that is actively being changed every minute by a developer. If there are two levels of testing, they can focus on different activity and not worry about messing up each other's data. Having an isolated performance environment allows you to run tests which may hang the machine, but if it's isolated, no other testers will be impacted.

When too many people try to do too many different things in the same environment, you end up with a lot of wasted time as one group waits for another group's test to finish so they can run theirs. And that wastes time, and wasted time can lead to wasted money, which Stargazer712 pointed out could make the strongest arugment.

Another reason (not as common) is data: if your application has sensitive personal data or credit card data, you usually cannot put that into test environments, and there is usually masking requirements for everything except the QA and Production environments.

Answer 6

You seem to have invested a great deal of effort in bringing about a cultural change in your workplace. This is a great achievement as change is difficult at the best of times, yet cultural change is not simply about changing people's minds, but about changing habits, breaking prejudices, and ultimately about opening potentially closed minds to greater possibilities. So the question to ask yourself at this point is "What did I miss?". The easy answer is that you may not have fully engaged with management.

Getting buy in from management is easy, but even harder is getting acceptance. Regardless of the arguments about money etc, the reality is that you need to be able to influence management's view of priority. Your manager will have a budget, and will want to show that budget has been applied sensibly and in keeping with company values and priorities. Some of those priorities will be fiscal, but others will be about serving other needs. In some cases, this may mean greasing the palms of other managers in order to get that promotion your boss has always wanted. In most cases though, it will likely be about finding ways to gain more business, or improve relationships with partners and customers. If you can't make your case in these terms, you will only be able to go so far before you find yourself at an impasse.

My suggestion is to try to make a case about productivity and how this affects budget, as others have suggested, but also to make the case in terms of your company's priorities and how your productivity might directly impact on the company's relationships with other companies.

Answer 7

Here's one: testability.

Having a testing environment gives you the freedom to perform tests on a database that would be inadvisable to perform in a production environment.

Answer 8

You want to change how your organization develops its software? Forget worrying about "reasons" for "doing things differently". Humans don't change behavior because of rational arguments. They change because of psychological influences on their habits.

So, where am I going with this?

While occasionally you can successfully change an organization's behavior through argumentation, there are other tactics that work better. These include:

• Grassroots support: Find ONE other "legacy" developer who is willing to give you a shot. Partner with him and change how things work. Don't announce the change. Just make the change. If anyone ever asks you about it, just say "Oh yeah, that's how we do it now."

• Take over responsibility. Volunteer to handle deployments for the legacy folks. Act like you love it. They may be happy to relinquish that responsibility. Then run it how you want to.

• Blame the right people for their mistakes. The next time a legacy app bug is introduced into production because of your stone age deployment mechanism, point it out. Do it subtly... Not in an email. Next time you're in a meeting with a manager, just casually mention the example of a reason that the deployment was problematic. "Yeah, remember how we were scrambling last Friday because of the Foo bug Bob checked into production? Yeah, that was a lot of wasted effort!"

• Make it easy to do it the better way. Look at the iphone, for example. There is ONE button on it. (Well, two). It's VERY easy to turn on. Make deployment-to-multiple-environment crazy stupid easy. Make it so easy all the managers can do it!

Answer 9

Its more of an issue when you start dealing with interconnected or legacy systems, systems the business depends on to be working and accurate. Its important because there needs to be segregation between the stages, its the reason why you dont DEV on PROD because it has the potential to cause millions of dollars worth of damage in lost time.

We always do DEV -> QA -> PROD (occasionally those steps are broken up into smaller pieces) with identical hardware behind them. Current production data is always pushed from PROD to QA to DEV.

DEV: is intended to be the development sandbox, where things are tried, iterated and beaten on any data in this environment should never be trusted and is regularly trashed by developers simply finding ways to solve a problem.

QA: Once your developers are satisfied with unit testing, its time for the test group to get there eyes on it. They run test cases, performance testing and find bugs. Those bugs/enchancements get fed back to DEV and the cycle continues until everyone is happy.

PROD: Once you get to this stage you should be sure the code works in conjunction with current data and your QA group/business users are happy with the implementation. If you did everything correctly you should simply be able to update the code and be done with it.

In the same way you would never release an untested product to customers you should never release untested code to a production environment.

If the company is not willing to invest the time to do it properly they will pay back the cost in emergency maintenance and errors 10 fold.

As a small example: We had one company who decided to make a change to a report in production on his own. No one knew that it changed until we came in to address a variety of issues a year or two down the line.

When we pointed out the irregularity in the report the CFO's face went white, turns out they were losing ~$250,000 a quarter because of someone making a quick change.

Happens more often then you think, if you cant afford to do it properly then dont do it.

Answer 10

Management has a big part behind the success of the Software Companies and Software Products which required to generate these environments. Let take an example of your project. If your software is developed on a large scale then if you do not manage your project requirements, process control, Test Builds etc then this is chances of failure. so that Project Management exist.

I am somewhat agree with @Stargazer712 that your statement point to the Money matters, But Check the following statement that i have gotten from Marc Hamilton's book on Software Development: Building Reliable Systems (Prentice Hall PTR, 1999, ISBN 0-13-081246-3). After looking all of these factors; my Opinion about your question is that Single Environment is not give you savings, it will make a long term process to complete the project/software. Distributed Environments will save Time and revenue as i learned and seen in my experience that what happened with the start-up software companies from which i have start my carrier.

There are lots of articles demonstrate that what matter for success, Check this one Organizing for Successful Software Development

Each individual in an organization has certain skills, and these skills are typically measured against formal or informal performance metrics leading to rewards (compensation) as incentives for future performance. The people in an organization establish its culture—those behavior patterns and values that are generally recognized as being adopted.

A great set of software developers will struggle and ultimately fail to meet their goals if they have to spend all their time fighting against an inappropriate organizational structure.

Many a software startup begins life with no more than a couple of developers working out of a garage. Not much organizational structure is required at this point in a company's history, but organizational structure still exists. For instance, in 1977, when Bill Gates and Paul Allen formed their partnership and officially named it Microsoft, the company had minimal organizational structure. Fewer than a dozen employees worked at Microsoft's first office in Albuquerque, New Mexico, and everyone knew who was in charge. No complicated organizational charts were needed to figure out everyone's reporting structure. At the same time, all employees knew their role in the company and what they were trying to accomplish. This was because any organizational structure that was needed could be informally communicated between each of the employees.

Answer 11

Forget time, money, testability, quality...how about reputation.

good, simple, irrefutable reasons for the separation of environments that would get managers lacking development experience to support this idea.

Uber recently shipped code to github that contained passwords for their live environment, allowing 'hackers' to download all their customer details. Uber says it was a breach, everyone else says"don't put the keys to your locks in public view. If their developers worked entirely on a dev environment, they might have released the keys to their dev environment on github, but that's entirely harmless. That the production ones were released shows how poor this idea of performing dev on the production environment is.

Just remind your manager that mistakes happen, so the way to avoid him being hauled before the CEO who is about to get a grilling in front of journalists and laughed at by the tech public is to take simple, obvious steps to prevent those mistakes being catastrophic ones.

Answer 12

Sounds like you have to many different environments and it is costs people to much time to setup an "environment".

You should have the least number of different "environments" that you can get away with, but be able to clone many copies for as many reasons as the you and the company desire (using "environment to mean system configuration)!

Optimally the only differences should be:

1. Size (minimal, recommended, biggest supported/tested against);
2. Staging and production don't have development tools
3. Production is protected against accidental overwrite of data
4. You can very easily load demo, test or [anoymized] clients data to development or staging servers

THEN the question of how much and what kind of testing should be done is a risk/cost business evaluation and decided on at a company level, because it is the business as a whole that will suffer if significant faults get out to a range of clients.

Later Edit: This prompted me to rationalize my naming conventions with my web products (thanks for the question). I have decided on four "environments", with testing split between qa (minimal single tier for testing functionality only) and staging (same architecture as production, for load/performance/volume testing).

The only real difference in provisioning is production/staging install a DB to a separate system, which I control by which groups the different servers are in. qa/dev have both the web server and db roles. Load balancing is done by cloudflare.

I also have an ENV_NO variable, which is passed onto the systems so I can choose to have as many "qa" or "staging" examples as I choose to.

So, to setup a second qa environment including my latest backup from live, the commands would be:

git checkout desired-branch/commit for provisioning
ENV_NO=2 bin/provision create qa
# come back after a short lunch

git checkout desired-branch/commit
ENV_NO=2 bin/cap qa deploy
# a minute or two

ENV_NO=2 bin/cap qa db:upload db:restore
# longer than I want once there is a decade of data ("longer" coffee break to traverse a 5.3km ADSL link)

Lastly, I have one extra (optional) server called "readonly" as the last safety net before hitting the ground. It is provisioned like a qa system but with last nights backup and update disabled (the software is also updated to last night).

It uses a "All eggs in a different basket" approach: It is provisioned with a different location / DNS registrar, DNS host, system host service providers. This is ultimate/last safety net so if everything has gone up in flames you can at least get to the data up to last night. The provisioning scripts isolate the difference between the different providers, so 99% is the same, just the readonly flag. Cloudflare load balancer will redirect traffic to the readonly site if all the live servers have failed.

Answer 13

When it comes to making a change, you will be lucky to have someone who will just listen to your professional opinion and implement suggested changes.

On my experience, each time I had to make a major change, I had to justify it in terms of savings the business will make. For example, introducing ReSharper into the development pipeline was pretty easy, as I was able to say something on the lines off:

ReSharper costs around £50 per developer. Average developer cost per year is £40k. ReSharper should increase developers productivity by at least 20% given its used to its full potential. Say developer spends 75% of their time actually writing code in IDE. 75% of 40k is £30k. £30k is now the cost of developer's productivity. An additional percent of productivity (1%) per year costs £300. To get additional 20% productivity, the business would have to spend £6k.

If you were to put this into perspective to the business, you can say that you can hire somebody else and get additional 20% productivity for £6k, or you can get the same result by spending £50 on a ReSharper license. Once the figures are in front of the business, the decision will be easy to make.

Now in regards to your questions with having multiple environments, all you have to do is find a way of calculating how much it costs the business every year to now have these environments.

You can ask your fellow developers to keep track of hours spent each week on configuring applications for development, deployment etc. For example, ten hours of senior developer's time might cost business £500. That's 10 hours that can be spent on development, or something much more important. You gather the figures for some period of time and provide business with a yearly cost.

I personally hate this kind of politics, but it's common and we have to live with it.