I'm creating a python module at work, and I have a config file with variables that change for dev or prod (i.e. S3 buckets). What's the best way to do this? Right now I have dicts in config with DEV or PROD options, but selecting this every time in the classes seems tedious (i.e. S3_BUCKET[DEV] etc).

  • Your code should not be aware of which environment it runs in, so I'm not sure by "selecting" your dict using constant you are doing the right thing...
    – Diane M
    Commented Feb 23, 2017 at 4:47
  • Open to suggestions about how to avoid that Commented Feb 23, 2017 at 4:59
  • Well, you have many options, but the most standard is most definitely to have two distinct config files. Load one conf file, and swap which you load depending of environment. That's the most standard way to get environmental variables such as db names, paths and all, but maybe you have a different requirement ?
    – Diane M
    Commented Feb 23, 2017 at 5:06
  • That seems needlessly complex: two files containing very similar variables. What about an environment variable grabbed at the start of the config file? Commented Feb 23, 2017 at 5:09
  • If you think that's complex I think you probably got my suggestion wrong. I will post an answer with a simple example. Again, I'm not sure that's what you need but generally speaking this is how I handle environments variables.
    – Diane M
    Commented Feb 23, 2017 at 5:11

3 Answers 3


When you need different values depending of the environment, there are several things you can rely on:

  • A command line argument such as --db-path=...
  • An system environment variable such as $MYAPP_DB_PATH
  • A fixed path local file (on unix systems, they are usually located at /etc).

Usually, I go for a fixed path local file, they are easier to handle in the program and can be extended easily. It's also always possible to add later on the flexibility to specify the path of that file by command line or system environment variable. The file should contain the variables of your logical environment. The code you run should never know which environment it runs in, because that would then mean you can't test your production code (which is a very bad idea).

This is a general advice and not particularily tied to Python, but this is an example of how you would do it the simple way in that language (i don't really know S3 buckets, I will just use plain json files for clarity purpose):


import json
with open('mycfg.json') as f:
    env_vars = json.loads(f.read())


    "path": "/tmp",
    "ip": "",
    "login": "root",
    "password": "root"


    "path": "/var/www",
    "ip": "",
    "login": "myapp",
    "password": "aunriseqvoa"

You would version the two files somewhere, encrypted if necessary (for example in an ansible vault). Be very careful of not commiting plain text production credentials in your repo since it will be hard to erase from git later on.

When you deploy, simply rename the one you need when running your module, or provide it to the run command, based on your implementation. Note that if that is practical, the cfg file can redirect to another data container (such as S3) if it brings any advantage over a plain file.

Generally speaking this is how I handle environment variables, but maybe I understood something wrong about your requirements.

  • If you run the prod and staging (or dev) on the same machine, how can you have both a 'mycfg.json' which is the production file and a 'mycfg.json' which is the dev file? Commented Jun 15, 2020 at 10:07
  • @BeChillerToo If you can stack several times an application on a machine, you cannot rely on a given path, and should then use either command-line arguments or unix environment variables to distinguish dev and prod configuration file.
    – Diane M
    Commented Jun 26, 2020 at 17:57
  • Yeah that's what I figured out, I'm using "--config=/path/to/my/config.json" to specify which config file to load. Commented Jun 27, 2020 at 12:25

I work on big enterprise application on top of Node.js. But I think, the approach can be the same:

Have config/default.py config. Here you can define common variables which are env independent. For instance:


config = {
    'app_name': 'Spotify',
    'jobs_num': 2,
    'host': 'localhost',
    'port': 3000

Then you can have specific configs for particular environments.


config = {
    'host': 'myapp.com'


config = {
    'app_name': 'Spotify',
    'jobs_num': 3,
    'host': 'localhost',
    'port': 3001

Then, I prefer always to have env.py config with the next content:


def get_env(var, default=None):
    return os.environ[var] or default

config = {
    'app_name': get_env('APP_NAME'),
    'jobs_num': get_env('JOBS_NUM'),
    'host': get_env('HOST'),
    'port': get_env('PORT')

And main: config/config.py which always should be imported instead of env specific configs:


from default import config as default_config
from development import config as dev_config
from production import config as prod_config
from env import config as envs

environment = os.environ['PYTHON_ENV'] or 'development'
environment_config = None
if environment == 'development':
    environment_config = dev_config
if environment == 'production':
    environment_config = prod_config

def chain_configs(*config_items): 
    for it in config_items: 
        for element in it: 
            value = element[1] 
                if value is not None: 
                    yield element 

### from python3.5

config = dict(chain_configs(*default_config.items(), environment_config.items, envs.items()))

This code has the next benefits

  • you can stick generic variables to files.
  • configs are regular .py files, so you can add dynamic expressions to them(if needed)
  • you are not restricted by your environment files, you can change the application by specifying env variables during the start. for example: PYTHON_ENV=production JOBS_NUM=3 python server.py
  • your application does not know in code, what environment it is - it is a good abstraction
  • clear priorities: env variables are the major. then goes env specific config, then default configs


  • Maybe, overloaded production way. To small scripts, I will prefer regular environment variables
  • chain_configs is not perfect, because it is not handling deep dictionaries
  • ability to specify env variables which are only specified in the env.py

It sounds from your comments that one part of your situation is that you have properties consistent through dev and prod that you want not to duplicate. One way of solving this is to have the stage (dev or prod) be part of your keys and use wildcards to have keys that are the same. For example

dev.s3bucket = "dev-bucket";
prod.s3bucket = "prod-bucket";
*.csvfile = "report.csv";

(You can adjust this approach for different config formats). You would then write a simple configuration manager that you can ask for "s3bucket" and it looks up the appropriate value for your environment.

The nice thing about this sort of approach is that it scales easily if you add another environment or you want to add even a completely different parameter to your configuration, such as having different resources when running in different regions.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.