2

I'm in the process of writing a development philosophy document for a small groups of developers (6.5 developers to be specific), but it's ideally a document that will set down the company's best practices as our team scales up. In the process of creating this document I've been interviewing each engineer to understand their development processes, to make something that's a good mix of helpful in eliminating redundant work, and not overly prescriptive.

Most of the issues I've found there are pretty good solutions for, but there's something that I assume must be a common issue that I can't seem to find a good solution for. That's managing environment variables that are required for local development. For instance, we've got a development key for AWS. It's required to use a lot of functions and features within the application we're developing.

At present the process is pretty bad. Obviously we can't commit an env file to Github as it's sensitive information that we don't want the entire org to have access to. Right now we have a secure S3 file that contains the env file required for development.

We've found that there are a lot of issues both keeping this up-to-date both in terms new env variables making it into this file, and pulling this down during development. It's not very surprising since it's a single piece of the development cycle that's not a part of Github, and it's something that only has to be updated or accessed pretty rarely. That said, when people run into issues it can be very frustrating since realizing that you've got the wrong S3 key, or something isn't a standard kind of error-checking.

Are there any solutions for managing environment variables that people have found work particularly well? We had an employee write a repo that is the best solution we've currently got (https://github.com/sihrc/privvy), but is there something better?

EDIT: The reason why private repositories are not the solution for this problem (we're already using private repos)

  1. These variables aren't limited to a single project. The development env file for instance is

  2. Version-controlling this file would make development difficult or impossible. If we roll an API key, that's the API key that we need to use forever more. If you need to revert to an older revision and the key changes, that's a problem

  3. It's not strange for us to have to add a third-party to a repo. It's VERY important that they be able to view the code, but not the credentials.

  • Easy: Don't use environment variables. They're not version controlled, they're stringly typed (i.e. they're not typed), and they're another dimension of configuration necessary to get a new machine going. – Alexander Jul 12 '17 at 18:54
  • @Alexander what would you suggest as an alternative? – Slater Victoroff Jul 12 '17 at 18:57
  • A version-controlled config file (XML, JSON, YAML or whatever), a config manager that parses it and lets consumers query it in a strongly-typed manner (i.e. returning an Int where appropriate, rather than a string containing Int. – Alexander Jul 12 '17 at 18:59
  • @Alexander that... doesn't solve my problem at all. The fact that it's an environment variable rather than an XML file is immaterial. It's about using privileged information in a development flow. Committing our AWS creds to git is exactly what we're trying to avoid. – Slater Victoroff Jul 12 '17 at 19:00
  • 1
    Totally disagree about versions control and creds. You definitely want a history of who changed what creds and when. What you should be doing is using a separate repo from the actual code to store the config files and ensuing that the access rights to the credential repo is on a need-to-know basis. During the build and deploy steps of your automated deploy system you should get the latest creds from the repo and combine them with the app code. Environment variables might be a good way to do that. But versioning credentials is very useful. – RibaldEddie Jul 12 '17 at 19:59
7

Obviously we can't commit an env file to Github as it's sensitive information that we don't want the entire org to have access to.

The fact that the information is sensitive doesn't mean it shouldn't be under version control. Similarly, the fact that you're using GitHub doesn't mean all your information is public. GitHub Enterprise has private repositories, and within your company, individual repositories can be restricted to a subset of employees.

Right now we have a secure S3 file that contains the env file required for development.

That's not the right approach. Instead of having your configuration in one place—the version control, you're putting information in two different locations, one being rather difficult to access. While S3 supports versioning, it's not as straightforward as in Git. Finally, unless you somehow configured S3 to use your corporate SSO server, you're forced to create additional accounts for every user who needs to access the S3 file. This is an opened door to severe security issues (like in “Hey, I can't access the S3 file. Seems my account has an issue again!”; “Well, just use mine. The password is ...”)

We've found that there are a lot of issues both keeping this up-to-date both in terms new env variables making it into this file, and pulling this down during development.

Obviously. As explained below, two sources increase complexity.

Are there any solutions for managing environment variables that people have found work particularly well?

Private repositories.

Another solution I've seen in several companies is to publish the settings into a public repository, but encrypt it. The problem with this approach is often the fact that the encryption key should be stored somewhere as well, and, more importantly, shared. This leads to two issues:

  • When a disgruntled employee has your keys, you have to decrypt and reencrypt everything with a new key, and share the new key to every concerned person.

  • Everyone shares the same key. It might be OK for a small team (although I don't see a reason why this would be the preferred approach), but will quickly become an issue if the key needs to be shared with outsiders: consultants, interns, or people from other teams.

Additionally, there's no need for version control of this file. Commits or diffs on this file aren't going to be meaningful

The first time you'll get your S3 file screwed by a disgruntled employee, you'll find that having a permanent history of every change is essential.

Same for diffs. Spending hours figuring out that someone replaced by mistake the wrong key, and then trying to figure out how to get the right keys back is much less fun when you can just track all the changes step by step.

The reason why private repositories are not the solution for this problem (we're already using private repos)

  • These variables aren't limited to a single project. The development env file for instance is

GitHub Enterprise has a branch restrictions feature. Although it is not as granular as directory-per-directory accesses, it may do the job.

Otherwise, you may switch from GitHub to a service which makes it possible to restrict accesses per directory. Personally, I'm using a SVN repository which, among others, contains the configuration of every virtual machine I deploy: one part is publicly available, but the other part which contains confidential data such as the secret keys to Amazon AWS, Google and Twilio services, is restricted to named persons. It consists of a simple directory.

  • Version-controlling this file would make development difficult or impossible. If we roll an API key, that's the API key that we need to use forever more. If you need to revert to an older revision and the key changes, that's a problem

I don't think that there is an actual solution which works in every case. For instance, what if a revision you made two days ago migrated from Bing Maps to Google Maps? Meanwhile, you may have removed all the Bing Maps API keys; therefore, if someone reverts to an older version, it won't work, and there is nothing you can do.

Otherwise, when it comes to the changes in API keys, this should be pretty rare. Unless you have a security breach, there is little use in changing those keys on regular basis.

  • It's not strange for us to have to add a third-party to a repo. It's VERY important that they be able to view the code, but not the credentials.

Same here. Branch restrictions in GitHub Enterprise, or per-directory permissions in version control systems which support this feature.

  • publish the settings into a public repository, but encrypt it. that just kinda pushes the problem back by one level. Now you have the same problem, but with regards to this key rather than the api keys (or other sensitive data) that it encrypts – Alexander Jul 12 '17 at 19:24
  • I think there's an important delta here, which is the difference between "the entire org" and the public. We're already using private repos. I'll add a couple of examples to the question since it's unfortunately not clear. – Slater Victoroff Jul 12 '17 at 19:27
  • @Alexander: exactly. This is why I don't like this approach. I listed it in my answer because this approach seems to be a current practice, but I don't encourage it. – Arseni Mourzenko Jul 12 '17 at 19:27
  • @SlaterTyranus: then what prevents you from giving access to the repository to only a team or some members of a team? – Arseni Mourzenko Jul 12 '17 at 19:27
  • @ArseniMourzenko See above. Often we need to give access to the repository to people outside the organization – Slater Victoroff Jul 12 '17 at 19:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.