I am developing a distributed application built as a collection of separate services which are served from multiple load balanced instances and trying to wrap my head around the distributed configuration problem.
As my background is embedded systems, and not so much about large scale SaaS systems, I'd appreciate some feedback on my ideas, and maybe even get some more ideas about implementing configuration the right way.
Some more background:
When I write "configuration" I mean user profile, which can be changed during run-time by an authenticated user using an API and be applied in a timely fashion to all instances (think email filter rules, as opposed to DB connection strings which are loaded and kept static for the lifetime of the instance and hidden from users).
Each service is implemented as a pre-forked HTTP server (for security and performance reasons) and separate services use separate configuration (meaning services do not generally know about other services' configuration structure/schema). Users change configuration relatively infrequently, and the requirement is that users will not experience service downtime due to configuration change (note that some deployments have exactly one instance of each service, so a gradual restart strategy is not really an option). Currently configuration is loaded from local INI files that are reloaded on signal, incurring a minor delay, which is acceptable.
So to my thoughts:
The plan is to use a central store (a DB or KV store such as consul), and having an agent running along each service instance to poll-for/receive changes, then call an endpoint locally that will validate the new configuration, rewrite the INI file and refresh in memory copy (meaning each instance manages the local configuration store for itself).
Why? This allows the services not to be dependent on any specific configuration provider/protocol/client, in addition there's no dependency on the central store being up, which in my case is mandatory, and also, no invalid configuration is being applied to any single instance.
Strategy to check that new configuration trickled: attach a unique value that is saved as part of the configuration, and having each instance export this value in a /health endpoint that a monitoring system can watch for (that does not necessarily mean that every pre-forked process updated it's own in memory copy, but let's leave this problem for another time).
How distributed configuration store is updated: A separate configuration service with an API that receives configurations for multiple services, having each of the configurations be validated by an arbitrary instance of the configured service, and only then persist in the central store.
Thanks head for any feedback or advice.