0

Problem

I have designed an evaluation tool (in python) and need some help to make it more user friendly. The tool requires ~100 (nested) parameters, which it gets from a yaml file and stores internally as dictionaries (e.g. one dictionary for data loading, another for preprocessing, ...). However, it is really difficult for the user to know which parameters are necessary due to the following two reasons:

  1. The input parameters change due to new feature requests. This sometimes breaks backwards compatibility for old yaml files.
  2. Depending on which kind of evaluation shall be performed and which input data is used, some parameters are either optional or mandatory. There are interdependencies.

Currently, the tool simply crashes mid-run if a necessary parameter is missing. Users then often have a hard time to figure out why the parameters are wrong (they may also suspect that the tool is broken). How can this be improved upon?

My solution ideas

I have the following ideas of my own but feel like there might be better approaches / architectures / design patterns out there.

  • Define a default parameter file, which is then overwritten by user parameters. Thereby, new params can be added without breaking backwards compatibility (already done).
  • Create reference yaml files, which are always tested in CI. However, lots of combinations are possible and the user would still have to manually check the reference yaml files...
  • Introduce a parameter-checking method at the beginning. Maybe via converting the dictionary to a set of named tuples which have the benefit of being immutable and throwing an error directly at the beginning instead of mid-run.
  • Your ideas...?

Thanks in advance!

1

The first thought I have here is that you might be able to solve your problem with better error handling. An approach I have used recently for a similar problem was that I use the get(key, default) method on dict for optional parameters.

For required parameters, you could simply use the dict[key] syntax and put a helpful error message in the except KeyError clause. If you have a lot of these, though, this can get a little ugly and tedious.

I haven't tried this out but one idea I have is that you could create a class to handle this. Let's say you have all text parameters. You could create an class that is str-like i.e. implements the methods that you use on these parameters. But all these methods would do is throw an exception with a helpful error message. You could construct these objects with the information that is required.

Another option that I think is probably better is to create a custom get(dict, key, default) method. Where no default is specified, you would again create a helpful error message.

If your goal is to provide documentation as well as information on failure, you might want to create a YAML file that mirrors the inputs that defines which elements are required (with a special keyword/text) and what the defaults are for those that are optional. This has a potential added benefit that it could be used to make the defaults configurable. You would then load this first and use that specification to process the configuration in one of the ways above. The 'defaults' file is also useful as documentation or as a source to generate documentation.

On a side note: if you are working with ~100 flat dicts in a file, you might want to look at TOML. You can do deeper nesting with it but I think where it shines is the broad, shallow tree situation.

| improve this answer | |
-1

So, after some more searching, I believe I have found tools for my needs. They parse YAML files and have the benefit of...

  1. ... throwing errors early on IF the yaml file is wrong
  2. ... providing default arguments
  3. ... being tied closely to the code (and thus git commits) in contrast to the yaml file.
| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.