4

I'm creating a python module at work, and I have a config file with variables that change for dev or prod (i.e. S3 buckets). What's the best way to do this? Right now I have dicts in config with DEV or PROD options, but selecting this every time in the classes seems tedious (i.e. S3_BUCKET[DEV] etc).

  • Your code should not be aware of which environment it runs in, so I'm not sure by "selecting" your dict using constant you are doing the right thing... – Arthur Havlicek Feb 23 '17 at 4:47
  • Open to suggestions about how to avoid that – user3730954 Feb 23 '17 at 4:59
  • Well, you have many options, but the most standard is most definitely to have two distinct config files. Load one conf file, and swap which you load depending of environment. That's the most standard way to get environmental variables such as db names, paths and all, but maybe you have a different requirement ? – Arthur Havlicek Feb 23 '17 at 5:06
  • That seems needlessly complex: two files containing very similar variables. What about an environment variable grabbed at the start of the config file? – user3730954 Feb 23 '17 at 5:09
  • If you think that's complex I think you probably got my suggestion wrong. I will post an answer with a simple example. Again, I'm not sure that's what you need but generally speaking this is how I handle environments variables. – Arthur Havlicek Feb 23 '17 at 5:11
2

When you need different values depending of the environment, you should probably go for a configuration file which contain the environment variables you need. The code you run should never know which environment it runs in, because that would then mean you can't test your production code (which is a very bad idea).

This is a general advice and not particularily tied to Python, but this is an example of how you would do it in that language (i don't really know S3 buckets, I will just use plain json files for clarity purpose):

Code.py:

import json
with open('mycfg.json') as f:
    env_vars = json.loads(f.read())
do_something(env_vars)

mycfg_dev.json:

{
    "path": "/tmp",
    "ip": "127.0.0.1",
    "login": "root",
    "password": "root"
}

mycfg_prod.json:

{
    "path": "/var/www",
    "ip": "192.168.1.44",
    "login": "myapp",
    "password": "aunriseqvoa"
}

You would version the two files in your repository (if it's private, otherwise don't version the prod one, to avoid a very serious and common security issue) and simply rename the one you need when running your module. The cfg file could maybe itself redirect to another data container (such as S3) if it brings any advantage over a plain file.

Generally speaking this is how I handle environment variables, but maybe I understood something wrong about your requirements.

1

It sounds from your comments that one part of your situation is that you have properties consistent through dev and prod that you want not to duplicate. One way of solving this is to have the stage (dev or prod) be part of your keys and use wildcards to have keys that are the same. For example

dev.s3bucket = "dev-bucket";
prod.s3bucket = "prod-bucket";
*.csvfile = "report.csv";

(You can adjust this approach for different config formats). You would then write a simple configuration manager that you can ask for "s3bucket" and it looks up the appropriate value for your environment.

The nice thing about this sort of approach is that it scales easily if you add another environment or you want to add even a completely different parameter to your configuration, such as having different resources when running in different regions.

1

I work on big enterprise application on top of Node.js. But I think, the approach can be the same:

Have config/default.py config. Here you can define common variables which are env independent. For instance:

config/default.py

config = {
    'app_name': 'Spotify',
    'jobs_num': 2,
    'host': 'localhost',
    'port': 3000
}

Then you can have specific configs for particular environments.

config/production.py

config = {
    'host': 'myapp.com'
}

config/development.py

config = {
    'app_name': 'Spotify',
    'jobs_num': 3,
    'host': 'localhost',
    'port': 3001
}

Then, I prefer always to have env.py config with the next content:

env.py

def get_env(var, default=None):
    return os.environ[var] or default

config = {
    'app_name': get_env('APP_NAME'),
    'jobs_num': get_env('JOBS_NUM'),
    'host': get_env('HOST'),
    'port': get_env('PORT')

And main: config/config.py which always should be imported instead of env specific configs:

config.py


from default import config as default_config
from development import config as dev_config
from production import config as prod_config
from env import config as envs

environment = os.environ['PYTHON_ENV'] or 'development'
environment_config = None
if environment == 'development':
    environment_config = dev_config
if environment == 'production':
    environment_config = prod_config

def chain_configs(*config_items): 
    for it in config_items: 
        for element in it: 
            value = element[1] 
                if value is not None: 
                    yield element 

### from python3.5

config = dict(chain_configs(*default_config.items(), environment_config.items, envs.items()))

This code has the next benefits

  • you can stick generic variables to files.
  • configs are regular .py files, so you can add dynamic expressions to them(if needed)
  • you are not restricted by your environment files, you can change the application by specifying env variables during the start. for example: PYTHON_ENV=production JOBS_NUM=3 python server.py
  • your application does not know in code, what environment it is - it is a good abstraction
  • clear priorities: env variables are the major. then goes env specific config, then default configs

Cons:

  • Maybe, overloaded production way. To small scripts, I will prefer regular environment variables
  • chain_configs is not perfect, because it is not handling deep dictionaries
  • ability to specify env variables which are only specified in the env.py

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.