Should I "modularize" my configuration file into different files?

Question

I have a simulation in Python which reads its configuration from a toml file. Since I have tons of parameters, the toml file can grow quite large.

This is an example file, similar in structure to my actual configuration file:

city.toml

name = 'New York'

[[houses]]
address = 'Foo Street 42'
color = 'red'
  [[houses.residents]]
  name = 'John'
  age = 35
  [[houses.residents]]
  name = 'Mary'
  age = 32

# [[houses]]
# etc ...
  # [[houses.residents]]
  # etc ...

Once loaded in Python, this turns into a dictionary similar to this:

city = {
    'name': 'New York',
    'houses': [
        {
            'address': 'Foo Street 42',
            'color': 'red',
            'residents': [
                {
                    'name': 'John',
                    'age': 35,
                },
                {
                    'name': 'Mary',
                    'age': 32, 
                },
            ],
        },
    ],
}

My issue is that the toml file can be quite repetitive. For example, if the user wants to simulate multiple residents identical to john = {'name': 'John', 'age': 32}, they have to go to each line where a house is defined, and copy-paste the value a bunch of times:

[[houses]]
    [[houses.residents]]
    name = 'John'
    age = 35

[[houses]]
    [[houses.residents]]
    name = 'John'
    age = 35

[[houses]]
    [[houses.residents]]
    name = 'John'
    age = 35

# [[houses]]
# etc ...
  # [[houses.residents]]
  # etc ...

which is both time-consuming and error prone. In particular, the user may run a simulation which is valid, but has undesired parameters, and only notice it later (if at all).

I'm thinking of solving this issue by "modularizing" the repeatable parameters. Something like:

# etc ...

[[houses]]
  # etc ...
  residents = ['John', 'Mary']

[[houses]]
  # etc ...
  residents = ['John']

[[houses]]
  # etc ...
  residents = ['John', 'Mary', 'James']

My idea is that, after parsing the main toml file, the Python code would be responsible for reading the strings in the residents array and load the respective residents from individual toml files, like:

John.toml

name = 'John'
age = 35

Mary.toml

name = 'Mary'
age = 32

The final Python dictionary would then be constructed at runtime, merging different "submodules" of the configuration file.

My questions:

• Is this a good pattern to follow? Is this approach actually used anywhere?
• Are there significant drawbacks? (One I can think of is passing a value like residents = ['Mark'] without an existing Mark.toml file - the code would have to deal with these situations somehow)
• Are there any alternative solutions that I have not considered?

Answer 1

This doesn’t look like a configuration file to me, but a database. So use a database.

Answer 2

Your problem is not whether to modularize a configuration file or not. Your problem is managing its content.

Independend from the structure - modularized or not - you should work on a UI to manage your configuration. The configuration may be human readable but better automatically generated.

Perhaps it is no bad idea as @gnasher729 suggested putting the data into a database. So it keeps the configuration part separated from the data part.

Edit:

To clarify:

• UI doesn't necessarily mean a GUI or a TUI. It could be a commandline interface. And as such helps the user to interact with the configuration in a safe/sane way.

• Database could be such simple thing as SQLite.

Answer 3

This is a reasonable way to approach avoiding repeating the same resident. You will need to ensure that each resident has a unique name you can use to refer to them, and (as you noted) ensure that each name actually exists, but doing this will mostly likely be a simple matter of raising and catching exceptions.

The only other draw back is that this might become burdensome when you have a resident that will only be included in one house since you will have to modify either multiple files, or the same file in multiple locations if you decide to merge the configs.

This sounds very similar to the way some network managers handle interfaces. You don't have to re-define each interface when routing traffic from a subnet through eth1, you only define them once and then refer to them by name.

All in all I think it largely depends on how common you think resident re-use might be. If re-use is more common than single-use the modularize, otherwise, leave it as is.

Answer 4

Can there be two different Johns?

The identifier must be the same as the filename, must it also be the same as the name?

In general, age is a really poor field to persist. Consider a date instead, and calculating age when needed.

As you want to modularize among others this to reduce duplication, consider allowing the other pattern at the discretion of the user. Sometimes, it it just overhead.

Answer 5

Your approach of separating house and resident definitions is a sound one, but managing such a large set of files is not really easy either.

What I would suggest is, while keeping the [[houses]] array as you proposed in your question, make a table [people] (in the same or a separate file), used like this:

[people.john]
age = 35

[people.mary]
age = 32

You still have a clear separation between house and resident definitions, while having everything in one or two places, making changing it much easier.

Missing references is something that you need to handle, but should not be hard. I my proposed people should become a KV (Python dict likely) with keys like john or mary.

The only downside here is that people unfamiliar with TOML could create residents with names which would break parsing - for example using a name like john.doe without quoting it, which actually creates a nested object.

Further reading: TOML's table specification.