0

I'm writing an application with a nested loop structure. Importantly, some of the iterations of each loop might be skipped if they have already been executed in the past and their results have been saved to disk. It goes something like:

For each dataset:
    If all models associated with this dataset have already been trained and saved to disk:
        continue

    For each random partition:
        If all models associated with this <dataset, partition> have already been trained and saved to disk:
            continue

        For each ML algorithm:
            If all models associated with this <dataset, partition, algorithm> have already been trained and saved to disk:
                continue

            For each hyperparameter configuration:
                If this <dataset, partition, algorithm, configuration> has already been trained and saved to disk:
                    continue

                Train model
                Save model to disk

I find the naive nested loop solution to be very repetitive, especially given the similarity between the "if" statements.

My current solution is to organize the execution into a tree structure. I have a root node responsible for iterating over the datasets. This root node has per-dataset children; the per-dataset nodes each have per-partition children; the per-partition nodes each have per-algorithm children; and the per-algorithm nodes each have per-configuration children. The per-configuration nodes are leaf nodes.

When a node is "executed", it will in-turn execute all of its "unprocessed" children nodes. A node is considered processed if and only if all of its children are processed. The per-configuration leaf nodes record their status on disk, and they're considered processed once they've completed training.

This is actually the second time I've run into a problem like this. Are there other design patterns that fit this type of problem?

New contributor
Alexander Guyer is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct.
3
  • 3
    Repetitive or not, this seems very easy to understand, and clarity is no small thing. Sep 22 at 13:55
  • @RobertHarvey I agree. I should mention that I'm printing progress bars for each nested loop level (using the tqdm Python package). Doing so requires determining certain statistics at each loop level, such as how many loop iterations have already been processed, how many need to be processed, etc. The implementations of the "if" statements are also somewhat complex. So there's quite a lot of repetition, and it gets a bit ugly. The tree structure allows me to encapsulate most of this repetitive logic in a couple of base node classes, which removes much of the repetition. Sep 22 at 14:11
  • This looks very much like a tree to me, so I don't see a compelling reason to look for something else. To fight the uglyness, refactor to small functions. For example, extract each loop into its own function. Extract each conditional check into its own function. Put the statistics at each loop level into its own function. Biggest problem here will be to find some consistent and self-describing names for all those functions (but good naming is always hard).
    – Doc Brown
    Sep 22 at 18:29

0

Browse other questions tagged or ask your own question.