What to consider in choosing an approach to dev environment creation for existing production system?

Question

Our team has not had dev and test environments since it started 2-3 years ago. We had decided to give this (creation of dev and test environments) more priority.

Current Environment

• One production environment with outdated software (OS, programming language, tools) - but security patches are regularly installed
• 5-6 developers connecting to the production machine to perform development
• Separate user in the OS that's considered to be the actual production environment where processes will run in an automated fashion
• Virtually no test suite

New Environment Options

We have (at least) two approaches to choose from, to set up the dev environment:

1. Clone production, write tests, upgrade everything

   1. Create a dev environment that resembles the current production environment as closely as possible
   2. Write tests
   3. Upgrade everything (may be gradually, one-by-one) and see if anything fails (and fix it)

   Pros: Less risky

   Cons: Slower

2. Create new environment with latest versions of everything, then migrate the code

   1. Create the dev environment with latest versions of everything (OS, programming language, tools, more standard practices)

   2. Start migrating the codebase (although the codebase has hundreds of scripts, it is possible to copy over 2-3 scripts at a time and ensure they run well in the new environment)

      1. Fix anything that doesn't work
      2. Write tests for the newer codebase

   Pros: Faster (probably); ability to take advantage of latest features sooner

   Cons: May end up being more time-consuming

Notes

• Some of the team members have been here for 2-3 years, while I joined a few months ago
• I'm biased towards option #2 (upgrade first, migrate code) - so, the pros and cons listed under both the options are results of my biased view
• On the other hand, one of my teammates is (I think) in favor of option #1 as it may seem a bit more conservative (so, less risky) - I may be wrong about this. He seems, in general, an open-minded person.

Answer 1

It might help to think of it as two different tasks - (a) the creation of a new development test environment and alteration of deployment practices, and (b) testing and implementing an OS upgrade of the production environment.

I'm inclined to think these are both major items of work in their own right. The overlap is only superficial in that they both involve configuring a test server - but spinning up servers is surely not the main struggle.

In (a) there won't be significant modification of existing code required, it's the team's future development and deployment practices - their habits - that will have to permanently adapt to the existence of two different environments. There might also be large changes (or new automations) needed in deployment tools and scripts, to cope with there now being more than one destination for code changes.

Meanwhile in (b), a change of production environment need not involve the permanent existence of a test server, but it does involve verifying and potentially revisiting a large amount of existing code to make sure it still works in the new environment. You've said there are virtually no existing tests, so there may need to be an extensive manual review and re-testing for this.

My suggestion is to tackle these two tasks separately, and whichever is started first, let the results bed in for a while without immediately embarking on the next.

If not, there's a danger of slipping into chaos by pursuing too many goals and changes at once, of timelines bogging down, and of the team becoming mentally worn out by planning and executing too many changes at once.