Better management of ~1000 lines of environment configuration properties?

Question

Background/Our situation:

Our environment has many components, including database, Windows Server, Linux, ETL platform, Scheduler Platform, and about 5 other types (so 8 total components = code repositories). I would say we have a fairly complex ecosystem with our application interacting with other applications in our enterprise.

Each component has its own source code repository, which triggers their own separate builds.

We have 50+ dev/qa environments and every other year or so we have to tear them all down or redo them somehow due to server/platform EOL and things like that.

Problem Statement I'm trying to help solve a problem for our environment management personnel (let's call them EnvOps for lack of better term).

Their responsibilities are the following:

1. Creating new environments
2. Updating environments (with latest builds), including production

So for each of these 50+ environments, we have 8 components per environment. For each component/environment combination, the EnvOps (and sometimes the developer if they're paranoid about making sure it's right) will check-in the environment configurations; we call these "environment properties" files. Doing the math, these are about 400 environment files for the EnvOps to manage. Some properties files have as many as 500 lines of values. Most average around 200.

The problem here is that creating these properties files are a drag and very tedious/cumbersome. I would like to reduce the strain for our EnvOps. I feel like we can't be the only ones having this problem, but I am failing to find patterns online (or I"m not searching the right way) for how to better manage these. The way we do it now is very error prone. When they create a new environment, what they do is just copy some stable environment's properties file (i.e. production) and then do a search and replace and type things in. To me this is error prone.

Another problem is when a developer adds a new key/value pair to a property file, the EnvOps is responsible for updating all the other property files or else something might break.

One thing I"m thinking of is to create a single tool that will take in 500+ inputs and then output out 8 files for a given environment - basically maven with a UI.

Is there any other ideas or solutions that you would do?

Answer 1

There is probably no silver bullet for this situation, but there are several tactics which can applied here to make the experience more smooth.

1. Reduce the redundancy!

   Maybe it is not possible to reduce the file count at all, but I guess in those 400 configuration files, huge parts are copy-pasted from each other. That's a problem!. Find parts / sections which are equal among different environments, and build some "include" mechanics to import those parts from a central place. That way, you will hopefully end up with one config file which contains 180 equal parameters (from those 200), and almost 400 (or at least 200 to 300) which contain only those 20 parameters which are not equal to the ones in the central configuration file.

2. A placeholder mechanics might be useful. Whereever the EnvOps have to search-and-replace for a specific value, think if that search-and-replace can be avoided by introducing some placeholder variable or template which can be filled somehow automatically. The variable could be set within the central config, or it could be something like a placeholder for the current machine name, or something completely different.

3. Follow the idea of "convention over configuration" to reduce the configuration efforts. Often some configuration parameters be determined automatically by deriving them from certain names of the environment, or from the value of other parameters. Make the system aware of such rules, then one does not have to maintain them all manually.

4. Automate, automate, automate: the idea of a tool which creates/generates the configuration files is not bad, but it should not force anyone to enter 500+ parameters manually. Such a generator makes most sense when it can generate a lot of properties correctly from only a set of small rules, which brings you to point 3 above. It is a useful solution when you are not able to embed the rule evaluation directly into the system which evaluates the configuration files.