Most of the time, I store development application config in root directory of the project, like this:

app
|-- config.json

But that doesn't seem to be the best approach, since this config ends up being stored in version control system - possibly resulting in leaked usernames, passwords and other sensitive stuff.

12 Factor App guide recommends dropping config files altogether and using environment variables for configuration setup:

... stores config in environment variables. Env vars are easy to change between deploys without changing any code; unlike config files, there is little chance of them being checked into the code repo accidentally; and unlike custom config files, or other config mechanisms such as Java System Properties, they are a language- and OS-agnostic standard.

That sounds really nice to me, but where does one store said environment variables, without checking them into source control? And what tools can I use to pass those variables to the app? There can be dozens of config options, and typing them by hand each time you launch the app is not nice - so they have to be stored in some kind of file somewhere. Said file thus will end up in source control, and we return back to where we started.

Is there some universally accepted way of handling configuration options, that doesn't have the risk of storing local configuration in source control?

  • 1
    Well, git at least has something like .gitignore where I can define files or folders that should not be checked into version control. As you say I don't see where Env vars should really help, eith you have a script to set them and should be stored together with the project or you have them 'somewhere' on your system (home directory or even in the machines startup scripts) which seems to create a whole lot of problems on its own, especially if a lot of configuration is necessary. In any case I would split the config files so that confidential information go in different files. – thorsten müller May 13 '15 at 8:26
  • @thorstenmüller - only problem with .gitignore that base/template config still has to be stored, which means that app has to read two configs - base one, with default options (stored in scm), and local one, that overrides base (and is not stored in scm). With env vars, I imagine that mass deployment gets easier - it's simpler to specify environment variables for new virtual machine setup than to write something to some non-standard file. – Rogach May 13 '15 at 8:37
  • Good question, After I started using normal user-specific application data store for this, life became easier ;-) – Wolf May 13 '15 at 10:12
  • 1
    Have you tried "Redis" redis.io . It is meant specially for the key-value structure storage only. – Karan May 13 '15 at 11:24
  • 2
    @jporcenaluk - I do like key-value storages, but adding a full-blown redis to application just to handle config management feels like a bit of overkill. On the other hand, maybe I never worked on big enough projects. – Rogach May 14 '15 at 13:07

Possibly there is no one good answer to this. It seems that you need to store this data somewhere safe, as it will be needed for disaster recovery purposes one day. This applies equally to properties files and scripts that set environment variables.

  • With the source code (in SVN/GIT etc) is a really bad idea, as this data will contain production database passwords and the like.
  • Your corporate nightly backup may be sufficient, but it is unlikely to keep a readily-accessible history of change.
  • The data needs to be versioned separately to the consuming software. In our current system, a change of configuration leads to a new application build, and this is just plain wrong.

We are currently looking at solutions to this problem, and are leaning towards a code repository with restricted access. This repository would contain cofiguration data only. Do others have experiences to share?

  • 1
    Having two separate repositories for one project seems like not a good idea - you can't do clean rollbacks or work with branches, because then you'll need to manipulate two repositories simultaneously (e.g. other branch requires some new config option, and when you switch to that new branch without also switching in config repository, things break in strange ways). – Rogach May 13 '15 at 9:23
  • 1
    @Rogach I take your point. There are valid reasons to keep some configuration with the code, but as you say in your question, sensitive stuff needs to go elsewhere. So two repositories seems unavoidable. Also, I did not mention that app servers often help here. Data sources and JNDI variables can be set up by the administrator and will not be public. – kiwiron May 13 '15 at 9:36
  • A second store makes sense. There maybe other kinds of data, also confidential, that may be stored along with the configuration (for instance production data that is under analysis to fix problems of customers). – Wolf May 13 '15 at 11:05
  • @Rogach They seem to attract a lot of hate, but git submodules would handle this just fine, I think – if the main one were set up right, and the restricted-access repo could just live inside it. – SeldomNeedy Mar 7 at 21:54

In examining problems and possible solutions, it helps me to use a method popularized by Jeff Atwood: If God were to create a way to store sensitive configuration information, how would he do it?

Well, he would know who needs configuration information and only give it to those people, and the information would never be able to be accessed by anyone else.

The first part should already be taken care of: your source control system should be authenticating users. And this approach is also given validity according to #10 in Troy Hunt's 10 Commandments of Source Control, "dependencies need to be in source control".

But how to keep it secure if it is leaked? Well, it doesn't need to be stored there in plain text! Use encryption. In .NET, there are steps you can take to encrypt connection string data in your config files. You would have to find the equivalent methods to do so with your particular technology of choice.

  • 1
    Just wanted to clarify - how encrypting configuration will help? As I understand it, you will be required to share the same decryption password between all the developers, and that sounds like calling for problems. – Rogach May 20 '15 at 11:02
  • If someone outside your company gains access to your repository, your passwords are obfuscated. If someone copies the files from a project onto a USB drive and leaves it somewhere, same thing. It will be more work to maintain, of course. More security usually comes at the price of convenience. This solution is a bit unwieldy, I'll give you that. I'm open to a better way to solve the OP's question! – jporcenaluk May 20 '15 at 13:52

Many people criticize storing configuration in regular files together with your source code but in my experience, this is actually a pretty good solution:

  • Simple to implement in any language. In many, you get support for complex configuration files out of the box. E.g. in the case of Java with Spring Boot, you get YAML support which can express any tree-like structure, and it's easy to have separate configuration files for different environments as well as a baseline config from which environment-specific files can inherit.
  • Configuration is needed to run your software, and changes to code often require configuration settings to be added/modified, so it's natural to keep configuration and code together.
  • Storing configuration with the source gives you all benefits of source control, like knowing who modified which setting and when or being able to check configs during a regular code review.
  • Unless you work for the CIA, the security argument seems overblown to me. So your database password is stored in a file on the machine where your app runs. Well, if someone get get access to the machine with your app, you're probably in a lot of trouble already - they can e.g. take down your app and start their own app in its place on the same port. In such a scenario, having access to the DB password might not be such a big issue. Unless all your connections are fully encrypted, having access to your machine, they can sniff much of the interesting data from network interfaces anyway.
  • You can use a tool such as Hiera to have a textual configuration file but not store passwords or other sensitive data inside it.

So, for many cases, textual configuration stored in source control together with the code is a good start.

If you are into distributed systems or want to be able to hot-swap your configuration without redeploying your applications, you may find a solution based around a configuration server better. Spring Cloud has support for such mechanisms, and the backend serving configurations can be a git repository or Eureka. You can also roll your own using e.g. Zookeeper. Any of these approaches will make it easier to manage consistent configurations on many servers to update configurations without having to rebuild and redeploy your software. This comes at a cost of course, which is learning the config server and how to use it from your applications as well as yet another system to deploy and maintain.

  • But the code changes hands to someone who doesn't own the secrets in the config files, there's going to be a real mess. – Tim Ludwinski Jan 12 '17 at 20:22
  • @TimLudwinski Keys / secrets belong to the company, not individual developers, so they should be maintained in such a way that they do not get lost if any single individual leaves. They might be issues and maintained by admins / security team for example, so that there is a central registry. – Michał Kosmulski Jan 13 '17 at 8:04

We are battling the same problem where I work. Right now all of our configurations are file-based and source controlled with the individual applications that use them. This leads to duplication and to the developers having access to production/qa passwords instead of just development.

That said I think we've come up with a good solution going forward. We are moving our config files to a separate git repo (labeled the config repo). We then set up a spring-cloud-config(java) server which simply serves the files from the config repo based on the profiles passed to it. This is great for Java applications which can use the client and download them at startup time. For our PHP/non-java apps we will pull down the file directly. (Not ideal). In the future we may write something that lets the PHP application download the configs on it's own and cache them somewhere, but it isn't high priority for the first run. I think of this solution as config-as-a-service which doesn't explicitly violate the 12 factor apps recommendations.

I believe zookeeper can be used for the same thing(I saw a setup with kubernetes+zookeeper) so I'm not quite sure why that answer got a -1 above.

Links:

https://spring.io/guides/gs/centralized-configuration/

https://cloud.spring.io/spring-cloud-config/

I think your options are somewhat defined by the OS you are deploying to

I would suggest, yes put the values in source control. BUT only the 'dev' versions. You want your source code to compile AND work! not include extra secret steps

Your build and deploy process should then swap these values out per environment during deployment. (octopus has this kind of model)

Instead of storing the whole configuration in one file, store it in several files.

  • Have a configuration directory. All files there are interpreted as configuration files, except maybe README*.
  • All file names are sorted alphabetically, and the files are loaded in that order. This is why files in such cases often start with a digit or two: 01-logging.json. 02-database.json, etc.
  • Data from all the files are loaded into the same configuration structure available to the application. This is how several files can complement each others' settings, and even override them in a predictable way.
  • Only store in the VCS the config files with safe-to-see values, or default values. Add the config files with secrets during deployment, or, better yet, use an authenticated secrets storage service.

On your nearest Linux box, take a look at /etc/sudoers.d or /etc/nginx/conf.d. It shows the same pattern.

Secrets management is a different beast. You can manage them as a manual step while you're small. You can use things like Zookeeper. You can even check the secrets into a VCS in encrypted form, and decrypt them as a deployment step. A number of other options exists.

(Also, an opinion piece: JSON is not a good config file format, because it does not allow comments; comments are crucial. TOML, YAML, and even INI formats are better in practical use.)

Apache zookeeper gives wonderful options to store the application configurations for distributed systems. Changes made at zookeeper can be captured and processed by having a curator or zookeeper listener at the application end.

  • 6
    What are the options? how does it work? Where does it store them? Is one preferred over another? What are the various advantages and disadvantages of each option? How does this interact on different operating systems? – user40980 Mar 8 '16 at 15:02
  • 1
    @Gangz -- I would be interested in a more detailed answer, please don't be discouraged by the downvotes, and improve your answer so it can be of help. – Jay Elston Mar 10 '16 at 19:15

protected by gnat Mar 24 '17 at 12:07

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.