Can "higher order function" feature allow/maintain abstraction and encapsulation?

Question

Below is the function repeat written using a functional paradigm, such that when called as repeat(square, 2)(5) it will apply the square function 2 times on the number 5, something like square(square(5)).

def repeat(f, n):
    def identity(x):
        return x
    def apply_n_times(n):
        def recursive_apply(x):
            return apply_n_times(n - 1)(f(x))

        if n < 0:
            raise ValueError("Cannot apply a function %d times" % (n))
        elif n == 0:
            return identity 
        else:
            return recursive_apply
    return apply_n_times(n)

def square(x):
    return mul(x, x)

With regards to abstraction, I see that repeat(square, 2) returns an implementation detail in the form of apply_n_times(n - 1)(f(x)) multiple times before providing the actual result.

With regards to encapsulation, for the expression f = repeat(square, 2) one could mutate the members of function object, for example: f.__name__='garbage'

Does the concept of higher order function allow supporting abstraction and encapsulation? Because they return the implementation details and provide access for mutation.

Such existing implementations in large software are very tedious to use, as the user has to have an idea of the implementation before using it.

Answer 1

Wrt abstraction, I see that repeat(square, 2) returns implementation detail in the form of apply_n_times(n - 1)(f(x)) multiple times before providing the actual result.

The function returned by repeat(square, 2) is not an implementation detail; it's the whole point of calling repeat. An implementation detail is something that the caller doesn't need to know about (and in most cases isn't allowed to know anything about either) and that could be changed without breaking any of the caller's code. The caller wants and needs the function returned by repeat.

It seems to me that you're looking at repeat as if its purpose is to give you the result of applying f, and that's why you see the fact that it returns apply_n_times as an implementation detail. If you only wanted the result of composing f with itself n times, you could have defined repeat to take three arguments. But what makes curried functions useful it the fact that you can make use of the in-between functions it creates! E.g.

numbers = range(1, 10)
numbersSquaredTwice = map(repeat(square, 2), numbers)

In other words, their purpose isn't to compute the final result, it's to compute a function that can compute the final result. The function returned by repeat is a useful thing to have in and of itself.

With regards to encapsulation, for expression f = repeat(square, 2) one could mutate the members of function object, for example: f.name='garbage'

Everything in Python has metadata that can be inspected and modified, and function objects are no exception. Python won't stop you from deeply inspecting and messing with every piece of data in the program. That lets you break almost any abstraction if you choose to do so. So you could say that it's Python that doesn't have very good support for encapsulation. In functional languages a function doesn't "know its name", because asking a function for its name makes about as much sense as asking an integer for its name. Functions don't need names, and if you choose to give them one, they don't need to know it.

But do note that every call to repeat produces a new function object, so even if you choose to muck around with it, the changes you made won't affect the return values of other calls to repeat.

Answer 2

The benefit of abstraction is that the caller doesn't have to know about implementation details. If I understand correctly, you're questioning this python construct because the caller can find about about implementation details. That's not the same thing.

Not having to know about how a method does its job is useful. It aids the software developer in their job, because you can think about one level of detail at a time instead of having to juggle two of them simultaneously.

Being unable to find out is something quite different. It can be useful to give the library programmer greater freedom for later optimization without anything breaking. But in practice, it suffices to simply declare in your API "Any intermediate types visible through the returned helper object are not part of the contract, and using them is undefined behaviour". That way the library writer retains the same amount of freedom, and only callers who willfully break the explicit contract suffer from it, while the main benefit (being able to ignore the details) stays unchanged.

Answer 3

The caller doesn't so much need to know the implementation details of repeat(f, n) as he needs to know its return value. In this case, the return value is a little more complicated than most, because it returns a function. In a static language like Haskell, this is easily documented in the function signature like:

repeat :: (a -> a) -> Int -> (a -> a)

This tells you it returns a function that takes an a and returns an a, but you don't need to know any implementation details of that function, just its signature and semantics.

The main weakness of dynamic languages like python is that their types, including their return types, are not documented in their function signatures (the main strength is the programmer isn't forced to document them). This leaves you with either documenting return types in a comment/docstring, or reading the function body or contextual clues in the calling code to determine them. You have always had to do this with all functions, not just higher-order ones. You probably just didn't notice because your functions have generally been small or simple enough to not be problematic, and complex third-party code you have used has generally had good docstrings.

Python also has a grand tradition of information being hidden by convention, not by dictate of the language. If you can't tell if something should be a private implementation detail or not, you need to document that as well. I generally consider the body of any closure to be a private implementation detail, whether returned or not.