Best practices for testing settings file with many parameters in simulation code

Question

I'm conflicted as to what is the best way to approach this problem.

I am writing a simulation in Python, which is parametrized by ~ 50 parameters. I have a JSON file where these parameters are set, so that it is simple to modify them and perform different simulations iteratively.

Snippet of settings.json (not actual parameter names):

{
  "global_settings": {
    "param1": 10,
    "param2": true,
    "param3": 0.5
  },
  "simulation_settings": {
    "param4": true,
    "param5": false,
    "param6": true 
  },
  "output_settings": {
    "file_settings": {
      "param7": "string"
      "param8": false
      "param9": null
    },
  "param10": true
  }
}

I've written a SettingsManager class whose job is to:

• parse the JSON into a Python dictionary;
• expose some parameters as attributes (so that objects using the SettingsManager instance won't need to access a 3-level deep dictionary entry);
• validate these settings, making sure that every mandatory parameter is present (by raising exceptions if they aren't).

Snippet of managers.py (where SettingsManager is located):

import sys


class SettingsManager:
    def __init__(self, settings_path: str) -> None
        self._data = self.parse_settings(settings_path)
        if self._data is None:
            sys.exit()
        self.validate_settings()  # calls all other validation methods

Snippet of main.py (responsible for creating a SettingsManager instance and running the simulation with the parsed settings):

from src.managers import SettingsManager
from src.simulation import Simulation


def main(path: str) -> None:
    settings = SettingsManager(settings_path=path)
    simulation = Simulation(settings=settings)
    simulation.run()


if __name__ == '__main__':
    main(path='settings.json')

The SettingManagers's validate_settings method calls downstream methods that go through each subditcionary of the parsed settings and make sure that the relevant keys exist (one method for one dictionary).

As you can imagine, writing code to validate all parameters was tedious. Still, I can't think of a better way other than checking each individual key in each individual dictionay/subdictionary, since each key has a different name, each subdict has a different structure and many different data types are used for values.

Now the real issue is testing: How do I test for the validation of the settings file?

Should I create test cases for every single key that could be missing? That sounds like a lot of work! Not to mention a lot of refactoring, since requirements are not fixed and parameters may be added/modified/removed down the road. I thought about making a mock dictionary in place of the resulting dictionary from the parsed JSON file, but I'm not sure that's beneficial. Am I not simply testing my mocked dictionary, then?

I'm struggling to find some resources about best practices when writing simulation code with many parameters. At the same time, I can't shake off the feeling that I'm missing something obvious here.

How do people deal with this many settings? Are simulations always painful to code? Should I just accept my fate and write tests for every single parameter? Should I completely drop the validate_settings method and just assume users (me + my colleagues) have a properly formatted settings file?

Thank you in advance for your thoughts, I'm very lost here.

Cheers

Answer 1

Consider to implement a more declarative approach for validate_settings. You wrote

Still, I can't think of a better way other than checking each individual key in each individual dictionay/subdictionary, since each key has a different name, each subdict has a different structure and many different data types are used for values.

I can think of one. I guess a generic description for the settings cannot be much more complex than your actual JSON file? Why not have a complementary description file in JSON for those values, for example, like this:

{
  "global_settings": {
    "param1": "integer",
    "param2": "bool",
    "param3": "double|optional"
  },
  "simulation_settings": {
    "param4": "bool",
    "param5": "bool",
    "param6": "bool|optional"
  },
  "output_settings": {
    "file_settings": {
      "param7": "string"
      "param8": "bool"
      "param9": "bool|optional"
    },
  "param10": "bool"
  }
}

Now, validate_settings can use this description and do the validation for data types and non-optional values in a generic fashion. And that makes the actual testing of validate_settings a whole lot simpler, since for getting full test coverage, you can now use a few simple tests which cover each data type once and the optional/non-optional feature.

Of course, you will also have to proofread the description file to make sure it contains no errors, but that is way less effort than writing a test for each original attribute (and proofread also there is no error in those tests).

As an alternative, you could also have a look at JSON schema, which seems to be way more powerful, but I am not an expert on this.

Answer 2

You don't have to test every value, as you're not testing the actual config. You're just testing the one "unit" which is the validate_settings method. Running this test won't tell you if your settings file is good, but it does tell you that you won't be able to start the program if your settings file is not good.

You haven't shared the code for validate_settings() but presumably it takes a dictionary of config values, and a list of mandatory values and makes sure that each mandatory value is in the config values keys.

In that case, I think you mock both the dictionary of config values, and the list of mandatory values. Check the positive case (that it allows a dictionary with the mandatory values) and the negative (that it disallows a missing value).

For what it's worth, you could've gotten away with using Python standard library ConfigParser.