Configuration in a Distributed Application

Question

I am developing a distributed application built as a collection of separate services which are served from multiple load balanced instances and trying to wrap my head around the distributed configuration problem.

As my background is embedded systems, and not so much about large scale SaaS systems, I'd appreciate some feedback on my ideas, and maybe even get some more ideas about implementing configuration the right way.

Some more background:

When I write "configuration" I mean user profile, which can be changed during run-time by an authenticated user using an API and be applied in a timely fashion to all instances (think email filter rules, as opposed to DB connection strings which are loaded and kept static for the lifetime of the instance and hidden from users).

Each service is implemented as a pre-forked HTTP server (for security and performance reasons) and separate services use separate configuration (meaning services do not generally know about other services' configuration structure/schema). Users change configuration relatively infrequently, and the requirement is that users will not experience service downtime due to configuration change (note that some deployments have exactly one instance of each service, so a gradual restart strategy is not really an option). Currently configuration is loaded from local INI files that are reloaded on signal, incurring a minor delay, which is acceptable.

So to my thoughts:

The plan is to use a central store (a DB or KV store such as consul), and having an agent running along each service instance to poll-for/receive changes, then call an endpoint locally that will validate the new configuration, rewrite the INI file and refresh in memory copy (meaning each instance manages the local configuration store for itself).

Why? This allows the services not to be dependent on any specific configuration provider/protocol/client, in addition there's no dependency on the central store being up, which in my case is mandatory, and also, no invalid configuration is being applied to any single instance.

Strategy to check that new configuration trickled: attach a unique value that is saved as part of the configuration, and having each instance export this value in a /health endpoint that a monitoring system can watch for (that does not necessarily mean that every pre-forked process updated it's own in memory copy, but let's leave this problem for another time).

How distributed configuration store is updated: A separate configuration service with an API that receives configurations for multiple services, having each of the configurations be validated by an arbitrary instance of the configured service, and only then persist in the central store.

Thanks head for any feedback or advice.

Answer 1

You touch upon one of the key problems with this kind of thing which is separating 'configuration' from 'application state'

There are tools such as https://www.consul.io/ designed to help with the dynamic changing of config on distributed services. But in my view that approach is a mistake.

I would define:

• Configuration : Settings loaded at startup and only change by redeploying the application
• Application State : Settings which are change dynamically by user interaction.

The problem with trying to dynamically change configuration is that you lose the change control around deployments and, if you don't replace that with some other system, are unable to say what the expected behaviour of your application is. Especially during a change over, or with cached settings.

Obviously you have the same problem with application state, but generally the application has been programmed with the idea that this changes dynamically in mind.

Maintaining a clear separation between these two types of state is critical if you want to avoid the kind of issues you mention.

In your particular situation, I would move the state from the INI file to a database. Now your API call can change the database and have the nodes call the same api to read the setting. This makes it totally clear that these settings are not the responsibility of your system. You can even add some sort of listen endpoint to push changes out to nodes.

A health endpoint which reports on the current application state is a great idea. These should be standard practice, reporting version number, config settings and connectivity checks will save you a tonne of investigative work when you get deployment problems.