Best practices for handling large number of structured configuration/property files

Question

Imagine a system that has a large number of servers. Each of them has a number of settings:

• Some specific to the server
• Some specific to the region
• Some common across all of them
• Maybe you can have some custom groupings, like this group of servers is for reading only
• etc.

Current practice I have in mind is a simple property structure with overriding abilities.

Lets take Google servers for the purpose of the example. Each one of them has a list of settings to load.

For example, the London server may have:

rootsettings.properties, europesettings.properties, londonsettings.properties, searchengine.properties, etc.

Where each file contains a set of properties and the loading sequence allows you to override properties, the further you go.

For example: rootsettings.properties may have accessible=false as a default, but is overriden in searchengine.properties with accessible=true

---

The problem I am having with this structure is it is very easy to get out of control. It is not structured at all, meaning you can define any property at any level and many items can become obsolete.

Furthermore changing a middle level becomes impossible as the network grows, as you now affect a very large number of servers.

Last but not least, each individual instance may need 1 special property, meaning your tree ends up with with a config for each server anyways, making it not very optimal solution.

I would greatly appreciate if you have any suggestions/ideas of a better configuration management architecture.

Answer 1

I think you need to ask yourself some questions first, and clarify some points, then you can better decide how to solve your problem.

First: who shall have control over the servers?

• Is it a single administrator who is going to control hundreds of servers? Then you need to centralize the configuration as much as possible.

• Or is each server potentially under control of an individual admin, who does not want his settings overruled or controlled by a centralized configuration? Then you should focus on decentralized configuration. If each admin has a handful of servers to manage at maximum, this is still manageable manually.

Second: do you really need tons of configuration options, or can you keep the number down to a few? Instead of making all and everything configurable "just in case", better self-restrict yourself to the options you know your system really needs. This can be done, for example, by

• making your software a little bit smarter (for example, what can the program determine automatically by asking the environment)?

• following "convention over configuration" rigidly - for example, by establishing certain naming conventions, or by deriving some options as a default from other options

Third: do you really want a hierarchical level of configuration hardcoded into the software? Imagine you don't know beforehand how many servers there will be, if a tree-like hierarchy really is the best structure for them, or how many levels the tree needs to have. The most simple solution I can think of is by providing no centralized configuration at all, only one config file per server, and let the reponsible server administrators decide themselves how they solve the problem of managing the configurations of multiple servers.

For example, the admins might write generator scripts which distribute a central config file to a group of different servers and making some minor modifications to each copy. That way, you do not have to make any assumptions about the "server distribution topology" beforehand, the topology can be adjusted at any time to the real world requirements. The drawback is, you need administrators with some knowledge of how to write scripts in a language like Perl, Python, Bash or Powershell.

Answer 2

I personally never liked config file inheritance. You mention having a loading sequence, not sure how that is defined or derived. Maybe it helps to make the sequence obvious by mirroring it in a directory structure you put the files in or including it in the file names.

rootsettings.properties
rootsettings.eurosettings.properties
rootsettings.eurosettings.londonsettings.properties

This will somewhat work up to a point where you have an aggregate of configuration values which do not align to a region. So you need to be considerate of the organisational axis you pick.

What I like more is isolating aggregates of configuration values into their own files and have a (leaf) configuration file point to one.

An example:

bigcity.searchsettings.properties
regional.searchsettings.properties

Inside londonsettings.properties you could have a value like:

searchsettings:bigcity.searchsettings.properties

This allows for more degrees of freedom or an additional one.

Answer 3

I had the same problem and open sourced my solution. You can find the source code here https://github.com/Bikeman868/Urchin

I use it for a large system with many servers, with several applications on each server, and multiple environments. It includes a UI written in Dart for managing the configuration.

You can contact me directly if you need help getting off the ground.

The solution I implemented is rules based. You generally start off with one rule that applies to all configurations, then add you can add rules like "all machines in this environment create log files to this UNC path". The rules can be specific to an environment, an application, a machine or an instance of an application. Rules can also be more specific, as in only this specific instance of this application running on this specific server.

Rules are applied in the order of least specific to most specific, and later rules can override the values specified in earlier rules. This means for example you can create a rule that applies to a particular application, then override it with a different value for a specific machine, or a specific instance of the application etc.

The server has a REST + JSON interface so works with most development systems, and also has a convenient client library for .Net applications.

Answer 4

These kind of settings are a nightmare to manage. Its best to try and reduce them as much as possible by having your software components handle all the cases.

ie instead of setting an api server up for a region, pass the region in with the api call and let a single server setup handle all regions.

However, given you are in the state you are in. I would advise against setting up a system for generating configuration settings. It only adds complexity to an already complex problem.

Instead, have the settings stored in your deployment tool per server type. (octopus deployment settings, teamcity file rewrites, etc) This allows you do be sure you are deploying the same config settings you did last time when upgrade the software and gives you fine grained control over changes.

You don't want to be in the situation where you make a change to your meta-config files and then have to test what config file that produces in your various regions/servers/custom groupings

Answer 5

No research; all personal opinion, 30+ IT experience. Sounds like a complicated data structure, that can change/modified quickly, with a large number of users.

Consider a database repository (eg.SQL) to structure the data, with custom extract processes that produces individual config files for each user. You can use a query of the database to determine the impact of a change before it is implemented; find duplicate occurrences, etc.. Individual flat files are part of the problem. Plus you can get change logs and recovery / rebuild support. With database control, you might be able to eliminate configuration inheritance problem. That type of solution will require additional database structure. The correct composite key structure is very important.

That's my suggestion.

You might look into some very expensive vendor system security and control systems that implement this type of service. Consider them. In general they will be environment dependent.