I have an application with an external configuration. Basically, there are a few basic environment variables, then the rest of the application's configuration gets pulled from AWS S3, is checked for validity, and then cached locally.
Given this is a PHP app, I can easily pass a query parameter to a page (ie. ?refresh_config=mySecretPassword) and make it reload the configuration at runtime. However, given the app is running in containers behind a load balancer, I'd need to reload configuration on every single instance. I've not come up with a way of doing this that I'm satisfied with. Only way I can think of is to store the configuration in a distributed cache (Elasticache/Redis) instead of local cache.
I'd prefer to keep the configuration cached locally for performance and reliability reasons. I'd prefer to not need to rely on an external source on every call to fetch configuration (Redis) if I can avoid it. Only thing I thought I could do to avoid using a distributed cache all the time is to check a distributed cache at intervals (every 2-3 minutes or something) and update the local cache from that if needed. In that case, configuration updates would not be immediately reflected, but soon enough.
Is my concern about pulling from distributed cache unwarranted? Performance-wise, I think I'm likely overreacting, one call to the cache to fetch the config wouldn't be a big deal. It's really the reliability I'm questioning myself the most about. Technically, we could always have a circuit breaker and if Elasticache goes down, we re-pull configs from S3 and store to local cache instead temporarily. Or I can make the app check only periodically for changes at some acceptable interval. Just trying to avoid more complexity if not really needed. I don't think reloading configuration will happen frequently, but I've had enough experience where it's useful to be able to flip a configuration setting temporarily and it's useful to be able to do it quickly, without having to fully restart apps, re-deploy containers, etc.
