Best way to handle dev/test/prod variables in Python?

Question

I'm creating a python module at work, and I have a config file with variables that change for dev or prod (i.e. S3 buckets). What's the best way to do this? Right now I have dicts in config with DEV or PROD options, but selecting this every time in the classes seems tedious (i.e. S3_BUCKET[DEV] etc).

Answer 1

When you need different values depending of the environment, there are several things you can rely on:

• A command line argument such as --db-path=...
• An system environment variable such as $MYAPP_DB_PATH
• A fixed path local file (on unix systems, they are usually located at /etc).

Usually, I go for a fixed path local file, they are easier to handle in the program and can be extended easily. It's also always possible to add later on the flexibility to specify the path of that file by command line or system environment variable. The file should contain the variables of your logical environment. The code you run should never know which environment it runs in, because that would then mean you can't test your production code (which is a very bad idea).

This is a general advice and not particularily tied to Python, but this is an example of how you would do it the simple way in that language (i don't really know S3 buckets, I will just use plain json files for clarity purpose):

Code.py:

import json
with open('mycfg.json') as f:
    env_vars = json.loads(f.read())
do_something(env_vars)

mycfg_dev.json:

{
    "path": "/tmp",
    "ip": "127.0.0.1",
    "login": "root",
    "password": "root"
}

mycfg_prod.json:

{
    "path": "/var/www",
    "ip": "192.168.1.44",
    "login": "myapp",
    "password": "aunriseqvoa"
}

You would version the two files somewhere, encrypted if necessary (for example in an ansible vault). Be very careful of not commiting plain text production credentials in your repo since it will be hard to erase from git later on.

When you deploy, simply rename the one you need when running your module, or provide it to the run command, based on your implementation. Note that if that is practical, the cfg file can redirect to another data container (such as S3) if it brings any advantage over a plain file.

Generally speaking this is how I handle environment variables, but maybe I understood something wrong about your requirements.

Answer 2

I work on big enterprise application on top of Node.js. But I think, the approach can be the same:

Have config/default.py config. Here you can define common variables which are env independent. For instance:

config/default.py

config = {
    'app_name': 'Spotify',
    'jobs_num': 2,
    'host': 'localhost',
    'port': 3000
}

Then you can have specific configs for particular environments.

config/production.py

config = {
    'host': 'myapp.com'
}

config/development.py

config = {
    'app_name': 'Spotify',
    'jobs_num': 3,
    'host': 'localhost',
    'port': 3001
}

Then, I prefer always to have env.py config with the next content:

env.py

def get_env(var, default=None):
    return os.environ[var] or default

config = {
    'app_name': get_env('APP_NAME'),
    'jobs_num': get_env('JOBS_NUM'),
    'host': get_env('HOST'),
    'port': get_env('PORT')

And main: config/config.py which always should be imported instead of env specific configs:

config.py

from default import config as default_config
from development import config as dev_config
from production import config as prod_config
from env import config as envs

environment = os.environ['PYTHON_ENV'] or 'development'
environment_config = None
if environment == 'development':
    environment_config = dev_config
if environment == 'production':
    environment_config = prod_config

def chain_configs(*config_items): 
    for it in config_items: 
        for element in it: 
            value = element[1] 
                if value is not None: 
                    yield element 

### from python3.5

config = dict(chain_configs(*default_config.items(), environment_config.items, envs.items()))

This code has the next benefits

• you can stick generic variables to files.
• configs are regular .py files, so you can add dynamic expressions to them(if needed)
• you are not restricted by your environment files, you can change the application by specifying env variables during the start. for example: PYTHON_ENV=production JOBS_NUM=3 python server.py
• your application does not know in code, what environment it is - it is a good abstraction
• clear priorities: env variables are the major. then goes env specific config, then default configs

Cons:

• Maybe, overloaded production way. To small scripts, I will prefer regular environment variables
• chain_configs is not perfect, because it is not handling deep dictionaries
• ability to specify env variables which are only specified in the env.py

Answer 3

It sounds from your comments that one part of your situation is that you have properties consistent through dev and prod that you want not to duplicate. One way of solving this is to have the stage (dev or prod) be part of your keys and use wildcards to have keys that are the same. For example

dev.s3bucket = "dev-bucket";
prod.s3bucket = "prod-bucket";
*.csvfile = "report.csv";

(You can adjust this approach for different config formats). You would then write a simple configuration manager that you can ask for "s3bucket" and it looks up the appropriate value for your environment.

The nice thing about this sort of approach is that it scales easily if you add another environment or you want to add even a completely different parameter to your configuration, such as having different resources when running in different regions.