1

I am working on a personal project for more than 6 months now, this project is composed of three distinct parts, simulation (wrapping a software), database related stuff (store the simulated data in a database) and modelization (a lot of machine learning related stuff)

To manage my config (paths and few constants) I started by putting all the variables into the config.py file at the root of the package (see below).

It worked surprisingly well. However there is a caveat, I generally work with simulation and database in tandem, or with database and modelization in tandem, the thing is, in this setup, I have all the path of the project depending on one experiment_name present in the config.py file.

This situation worked surprisingly well when I was working only on database and simulation, but now the project is growing and it is not that good anymore.

Now I would need to be able to change the experiment_name when launching a test, however I like the fact that I can import all the path and constant by just doing import config as c, (I can then do c.path.. and know where does the variable come from).

I am searching a new way to handle the paths and config variable in my project. I am thinking about putting all the config file into a class and do something like c = Config(experiment_name), I have two questions related to that:

  • Is it a good idea/good practice ? Is there a better option ?
  • Where should I create the object inside the core files ?

config.py

import socket
from pathlib import Path, PurePath

experiment_name = 'experiment_0'

enable_simulation = True
development_hostname = 'DESKTOP-VXXXX'


path_project = Path(__file__).parent.parent

path_experiments_folder = path_project.joinpath('experiments')
path_experiment_folder = path_experiments_folder.joinpath(experiment_name)
path_experiment_input_config = path_experiment_folder.joinpath('input.json')
path_experiment_output_config = path_experiment_folder.joinpath('output.json')

path_databases_folder = path_project.joinpath('databases')
path_experiment_database_folder = path_databases_folder.joinpath(experiment_name)

if socket.gethostname() != development_hostname and enable_simulation:
    run_simulation = True
    path_simulation_root_directory = PurePath('/home', 'thomas', 'FOLDER')
else:
    run_simulation = False
    path_simulation_root_directory = path_project.joinpath('FOLDER')

state = f"hostname {socket.gethostname()}, simulation {['OFF', 'ON'][run_simulation]}"

simulation/core.py

from .. import config as c
from utilities import sub_function


def function_that_i_call_in_my_scripts(path_experiment_input_config=None):
    path = path_experiment_input_config or c.path_experiment_input_config
    sub_function(path)


class ClassThatICallInMyScripts:
    def __init__(self, path_experiment_input_config=None):
        self.path_experiment_input_config = path_experiment_input_config or c.path_experiment_input_config

    def foo(self):
        return  # do something

simulation/utilities.py

def sub_function(path):
    print(f'do something in {path}')

idea of new config.py

import socket
from pathlib import Path, PurePath


class Config:
    def __init__(self, experiment_name, enable_simulation=True, development_hostname='DESKTOP-VXXXX'):
        self.experiment_name = experiment_name
        self.enable_simulation = enable_simulation
        self.development_hostname = development_hostname

        self.path_project = Path(__file__).parent.parent

        self.path_experiments_folder = self.path_project.joinpath('experiments')
        self.path_experiment_folder = self.path_experiments_folder.joinpath(experiment_name)
        self.path_experiment_input_config = self.path_experiment_folder.joinpath('input.json')
        self.path_experiment_output_config = self.path_experiment_folder.joinpath('output.json')

        self.path_databases_folder = self.path_project.joinpath('databases')
        self.path_experiment_database_folder = self.path_databases_folder.joinpath(experiment_name)

        if socket.gethostname() != development_hostname and enable_simulation:
            self.run_simulation = True
            self.path_simulation_root_directory = PurePath('/home', 'thomas', 'FOLDER')
        else:
            self.run_simulation = False
            self.path_simulation_root_directory = self.path_project.joinpath('FOLDER')

        self.state = f"hostname {socket.gethostname()}, simulation {['OFF', 'ON'][self.run_simulation]}"

Structure of the project, many paths depends of the experiment_name

├── package
│   ├── __init__.py
│   ├── config.py
│   ├── simulation
│   │   ├── core.py
│   │   └── utilities.py
│   ├── database
│   │   ├── __init__.py
│   │   ├── initialization.py
│   │   ├── insert.py
│   │   └── retrieve.py
│   └── modelization
│       ├── __init__.py
│       ├── preprocessing.py
│       └── training.py
├── tests
│   ├── test_simulation
│   │   ├── test_run_simulation.py
│   │   └── test_multiprocessing.py
│   └── test_database
│       ├── test_initialization.py
│       ├── test_insert.py
│       └── test_retrieve.py
├── databases
│   └── experiment_0
│       ├── train.db
│       └── test.db
├── FOLDER
└── experiments
    └── experiment_0
        ├── output.json
        ├── input.json
        └──  source
            └── lot_of_files_for_simulation

1 Answer 1

0

I personally would use a function that returns a dictionary instead of a class in config.py, e.g. like this:

import pathlib

def configure(experiment_name):
    config = {}
    config["simulation_base"] = pathlib.Path("~/simulation")
    config["experiment_base"] = pathlib.Path("~/experiments", experiment_name)
    return config

Then, when you use it like this:

from config import configure

my_config = configure("important experiment")

Why would I do it like this?

  • The idea of a class is to encapsulate data and functions that use/manipulate that data. This does not seem to apply to your problem.
  • A function maps better to the concept some data (experiment_name) in -> some data (config) out.
  • A dict seems to provide all the functionality you need from your config.
  • No need to type self all the time ;)

If you want to keep the .key access, you could simply convert the dict to a named_tuple before returning it:

from collections import namedtuple

MyConfigTuple = namedtuple("MyCnfig", my_config)
my_config_as_named_tuple = MyConfigTuple(**my_config)
print(my_config_as_named_tuple.simulation_base)

In any case, the most important thing to consider in your case is probably: What will the me in 2 years want to change? And, is it worth to change the class I have already written to use another approach?

1
  • That seems such a good idea ! I implemented the day before the "idea of config", and it work quite well, but I think I'll certainly switch the class to the combination of function/dict :) Moreover the namedtuple approach is quite nice ! Thank you very much :)
    – th0mash
    Feb 27, 2021 at 11:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.