Design Patterns for Passing Large Quantity of Parameters in Machine Learning

Question

I am looking to better understand best practices for handling large quantity of parameters. I am particularly interested in the types of parameters involved in machine learning code bases and considering the patterns in which they're manipulated. These machine learning parameters typically possess the following characteristics:

1. Do not change: Once the configuration is specified, the code does not change the value.
2. Name collisions: Two modules that share the same parameter (e.g. lr for learning rate).
3. Defaults exist: Many times a specific run of the code only requires configuring a handful of parameters, the rest using defaults.
4. Related parameters can be pervasive: For example, the minibatch_size may be used by a large quantity of functions, including data preparation, gradient computation, etc.

I have leveraged three patterns in my experience:

Pattern A (Passing using Kwargs)

def do_b(c=3, **kwargs):
    b = kwargs['b'] + 1
    a = kwargs['a'] + 2 # Raises KeyError: 'a'
    return b

def do_a(a=5, c=4, **kwargs):
    b = do_b(**kwargs)
    a = a + 1
    return a, b, c

def main():
    a = 1
    b = 2
    c = 3
    do_a(a=1, b=2, c=3)

if __name__ == "__main__":
    main()

This pattern is the most clunky in my experience for several reasons:
1. Unpacking a variable from kwargs in caller function make it unavailable in the called function
2. Where are the defaults set?: If the caller does not override the defaults specified in the function's parameters, you could be traversing a large code base trying to find where the default is set. Also, the default may be specified in several different functions, making it difficult to determine which default is used (e.g. c in the example).

Pattern B (Passing via Global Object)

This pattern treats the configuration parameters more or less as global variables. The Config class is instantiated only once and the object gets passed around from function to function.

class Config:
    def __init__(self):
        self.a = 1
        self.b = 2

def do_b(opt):
    a = opt.a + 1
    b = opt.b + 1
    return b

def do_a(opt):
    a = opt.a + 1
    b = do_b(opt)
    return a, b

def main():
    opt = Config()
    do_a(opt)

if __name__ == "__main__":
    main()

Pros
1. Enables easy access to any configuration parameter as long as the configuration object is within the namespace.
2. Enables methods to be leveraged for controlled updating of configuration (e.g. merging configurations).

Cons
Global variable concept and all of the cons that come along:
1. Any function can update the object, and therefore makes it difficult to trace down the bug
2. No argument/parameter checking at compilation time.

Pattern C (Closures)

I have also seen each module, and their respective configuration parameters, exposed at a high-level using closures via lambda functions. See below for an example from ShangtongZhang/DeepRL:

class DDPGAgent:
    def __init__(self, config):
        self.network = config.network_fn()
        self.replay = config.replay_fn()
        # ...etc

# DDPG
def ddpg_continuous(**kwargs):
    config = Config()
    config.merge(kwargs)
    ...
    config.network_fn = lambda: DeterministicActorCriticNet(
        config.state_dim, config.action_dim,
        actor_body=FCBody(config.state_dim, (400, 300), gate=F.relu),
        critic_body=TwoLayerFCBodyWithAction(
            config.state_dim, config.action_dim, (400, 300), gate=F.relu),
        actor_opt_fn=lambda params: torch.optim.Adam(params, lr=1e-4),
        critic_opt_fn=lambda params: torch.optim.Adam(params, lr=1e-3))

    config.replay_fn = lambda: Replay(memory_size=int(1e6), batch_size=64)
    ...
    run_steps(DDPGAgent(config))

And then the classes are instantiated by the DDPGAgent. Unfortunately, this couples DDPGAgent with the lambda functions and makes testing very difficult.

Which of the three patterns would you chose and what are the pros and cons? Do you have any better options? Any guidance is greatly appreciated!

Answer 1

Yes, Yes, and Yes

You have struck on all the ways for passing information around.

• Construct a container and refer to it everywhere
• Pass many, many arguments through the function signature
• Construct an Agent and have them solve retrieving the information based on a request.

There really isn't a best way.

What might be reasonable will depend on the platform, language, and architecture of the program.

---

1. Do not change: Once the configuration is specified, the code does not change the value.

... It is certainly possible to make memory read-only. However I don't think this is the goal you are actually looking for.

I think what you mean is that a given function/object respects the configuration information provided to it and acts properly.

That is achieved by writing code in the function that does just that. If it passes the value through to some collaborator, then that is what it does. If it performs some computation then it does that. This is shown to be working by using unit tests that show the behaviour is as expected from the various configurations.

If each piece respects the passed in configuration, then the sum of the pieces will respect the configuration.

2. Name collisions: Two modules that share the same parameter (e.g. lr for learning rate).

Name collision is easy enough to solve. The trick is to add a prefix, a suffix, or use an indexor.

A naive prefix/suffix would be to just add more letters to the name, eg: foo_lr or lr_foo. A more elegant approach is to nest the configuration. Instead of having a flat container with whatever key/value pairs, you instead have a tree. The path through the tree adds prefixs/suffixs to the variable name until its sufficiently nuanced, eg: foo.lr or lr.foo. The beauty of using a tree is that it can be decomposed into sub-trees and even smaller sub-trees as more and more specific functions are called.

Similarly you could use an indexor, by creating something like an array. So lr is actually lr[index] and index identifies the specific context. The beauty here is that lr and other variables such as w can be made to have the same index as the node they are associated with.

3. Defaults exist: Many times a specific run of the code only requires configuring a handful of parameters, the rest using defaults.

It usually a bad idea to default values. If you must it is a better idea to have a factory which accepts your small set of parameters and then fleshes them out into the full high detail configuration expected by the rest of the code.

That way you can create a different set of defaults by creating a different factory, and none of your already working code is affected.

4. Related parameters can be pervasive: For example, the minibatch_size may be used by a large quantity of functions, including data preparation, gradient computation, etc.

Again, What might be reasonable will depend on the platform, language, and architecture of the program.

• An Object Orientated architecture would co-locate this data with the set of functions that share it.
• A pure functional program would have the value passed in as an argument, it should not care how the argument is sourced.
• An Actor program would expect the information to be passed in by request, or be retrievable by request.

Your rough choices are:

• keep one/a few copies of the data and share a reference to it.
• copy the value around through different configuration objects/agents/arguments

Neither way is superior.

• The more shared the data is, the more shared its interface is, and therefore more likely to cause issues when the code is changed, but updating data is far easier.
• The less shared the data is the less likely that a change will ripple through the code base, but also the more difficult it is to change the data consistently, and the greater change of inconsistency.