3

In this toy example some_function is... some function that takes a dict as an input and modifies it in place somehow.

def some_function(dct: dict):
    """Do something to the items in the dictionary.

    Args:
        dct: A dict that ... some description.
    """

I happen to know the caller of this function will often (but not always) have a list of dicts to do this with.

The caller could do:

ls_dct: list[dict]
for dct in ls_dct
    some_function(dct)

but this happens so often that I wanted to make it easier. I could do something like:

def some_function_with_maybe_loop(dct_or_dcts: dct | list[dct]):
    """Do something to the items in the dict(s).

    Args:
        list: A dict or list of dicts that ... some description. If there is a lists of dicts, all the dicts are
            processed independently in a for-loop.
    """
    if isinstance(dct_or_dcts, dict):
        dct_or_dcts = [dct_or_dcts]
    for dct in dct_or_dcts:
        # do something

but I have lots of these sorts of functions and I don't want to have to repeat that logic. So instead, I could make a decorator:

def maybe_process_multiple_dicts(f: Callable[[dict], None]) -> Callable[[dict | list[dict]], None]:
    """Decorator that takes a function that works with a single dict and makes it work with a list of dicts.

    This works by wrapping the function in a for-loop over the dicts.
    """

    def wrapped(dct_or_dcts: dict | list[dict]):
        if isinstance(dct_or_dcts, list):
            for dct in dct_or_dcts:
                f(dct)
        else:
            f(dct_or_dcts)

    return wrapped


@maybe_process_multiple_dicts
def some_function(dct: dict):
    """Do something to the items in the list

    Args:
        list: A dict that ... some description.
    """

But now I have a dilemma. The docstring for some_function and the argument name are both wrong. They should match that of some_function_with_maybe_loop. So should I update the docstring to reflect the modification imposed by the decorator? And supposing the answer is yes, then what should I do with the argument type-hint. For the type-hinting to all work it should stay as just dct: dict, but for the semantics to work it should be dct: dict | list[dict].

4
  • 1
    Remember that the decoration effectively does some_function = maybe_process_multiple_dicts(some_function), so the resulting value some_function has no docstring and its signature is dct_or_dcts: dict | list[dict]
    – jonrsharpe
    Jul 5, 2023 at 14:08
  • @jonrsharpe oh that's even worse. Then the decorating function would have to adopt the decorated function's documentation and append some of its own. Jul 5, 2023 at 15:33
  • 3
    Yes, that's what functools.wrapped is for.
    – jonrsharpe
    Jul 5, 2023 at 15:38
  • 3
    Honestly, I would recommend not accepting a single dict or list of dicts. Define a function that takes a list, and the caller can pass a singleton list if necessary. Your decorator is then little more than the built-in map function/class.
    – chepner
    Aug 4, 2023 at 12:13

2 Answers 2

2

You're asking the wrong question. Or trying to solve the wrong problem.

Every tool is a good fit for some set of problems. You're trying to use @decorators in a situation where they're not appropriate.

A glance at either PEP-318 or stock examples of decorators shows they're a good fit for functions that have substantially the same signature. Common examples include caching the result, enforcing a timeout, or logging the elapsed time.

What is wanted in this situation is the ability to quickly make many functions accept one-or-many arguments. We already have a common "grab the arg" idiom for avoiding mutable defaults:

def foo(d: dict = {'my': 'default'}):
def foo(d: dict | None = None):
    d = d or {'my': 'default'}

one-or-many

Rather than a one-line decorator, you want a one-line "grab the arg" idiom for your various functions. Define a helper for that.

def foo(d: list[dict] | dict) -> None:
    ds = make_list(d)
    ...

def make_list(d: list[dict] | dict) -> list[dict]:
    if isinstance(d, list):
        return d
    return [d]

Now foo has a nice uniform list of dicts to worry about.

If you wish to wrap a map(foo, ds) loop around the list, that may involve another trivial helper, and perhaps renaming each "singular" function to be _private(), with only the more flexible "one-or-many" function being in the public namespace.


Your library routines can certainly take on this responsibility.

But consider making it the responsibility of each caller. And then audit the call sites with mypy. Turning foo(d) into foo([d]), or foo(make_list(d)), isn't a huge burden on the caller, and it can simplify the code. Less automagic tends to mean less room for developer misunderstandings and less room for bugs to creep in.

1
2

This is a tough situation where no answer is completely right.

I would definitely keep the dict annotation, because otherwise it doesn't match the behaviour of the function it is attached to, maybe with a comment explaining how that annotation doesn't match the annotation of the decorated function.

In terms of documentation, that should document how the function is used, not how it is implemented. So as counter-intuitive as it may be, I would suggest something like this:


@maybe_process_multiple_dicts
def some_function(dct: dict):
    # NOTE: the decorated function also accepts a list of dicts, which calls this function multiple times
    """Do something to the items in the dict or list of dicts

    Args:
        list: A dict or list of dicts that ... some description. If there is a lists of dicts, all the dicts are
            processed independently in a for-loop."""
1
  • "In terms of documentation, that should document how the function is used, not how it is implemented. " - This!
    – Doc Brown
    Apr 2 at 6:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.