2

I was just wondering this while programming in python. If I have a function foo which take a collection (list or tuple, perhaps set etc.) and manipulates on its elements somehow, e.g. filter, trim, increases, etc. should such function return the same type? For instance, if it accepts any collection and filter things out, what should it return as an output object? Is it important at all?

There are also other containers, like numpy. So I have a numpy object a. There might be operations I cannot do on a using methods from numpy and therefore have to convert it to a list. Should I check for a type and return the same type?

If yes, is there a general technique to check type and return the same one?

  • 1
    Your question would be a better one if you had a concrete example of what you're talking about (and not a FooBar one either). – Robert Harvey Jan 18 '18 at 15:54
3

It isn't generally possible to return a new collection, because you don't know what the constructor expects. I.e. type(original)(new_values) may or may not work.

Instead, you should use the most general collection Python knows: an iterable. In particular, this allows you to conveniently use generators.

Compare this function returning a list …

def only_positive(items):
  out = []
  for x in items:
    if x > 0:
      out.append(x)
  return x

… with a generator-based function:

def only_positive(items):
  for x in items:
    if y > 0:
      yield x

Note that many builtins that previously operated on lists where changed to generator function in Python 3. E.g. for Py2, filter() is documented to return:

If sequence is a tuple or string, return the same type, else return a list.

But in Py3, filter() is actually an iterator object.

Numpy takes a different approach because the whole point is performing operations on a whole array. Typically, these functions either

  • take and return a single scalar value where the computation is performed as for a single-element array, or
  • take an ndarray or an array-like object that can be coerced to an ndarray, and return an ndarray.

The input type is not generally preserved.

| improve this answer | |
  • So if I want to return the most general collection in Python, iterable, shall I define something in a function like a class with required methods like iter, next, etc? Sorry hopefully you understand if my question is too trivial and answer is obvious. It is still not clear enough for me :/ – Celdor Jan 18 '18 at 21:45
  • @Celdor that's one way, but generator functions are much easier. Just yield some value when it's ready, and let the caller worry about whether they need to store it in some data structure and what that data structure would be. E.g.: result = list(your_function(1, 2, 3)) – amon Jan 18 '18 at 21:58
2

There are generally two kinds of collection frameworks: those without duplicate code but losing the collection type, and those which preserve the collection type but have duplicated code.

Most fall into the first camp, e.g. in .NET, everything returns IEnumerable, in Ruby, everything returns Array, in Python, everything returns an iterable, etc. Smalltalk falls into the second camp: all operations preserve the type, but at the cost of simply duplicating the operations for every type.

Scala has the first large-scale collections library I know of that manages to be type-preserving without duplication. The key idea is the concept of builders, which are objects that know how to (efficiently) build collections of a specific type. So, an operation like filter only needs to look up the builder for the collection type and then it can use this builder to efficiently build the result collection without actually having to know about each and every collection there could ever be. In Scala, it is the type system (or more precisely implicit resolution) that takes care of this lookup at compile time (so that there is no runtime overhead), but it can just as well be done at runtime.

Note, however, that type-preserving collection operations may have surprising consequences. For example, you don't expect map to change the length of a collection, but

map(Set(1, 2, 3, 4), lambda el: is_even(el)) == Set(true, false)

Also, in some cases, you actually have to return a more general type, or another related type. E.g., a ByteSet can only contain the numbers 1-255, so

map(ByteSet(16), lambda el: el.__str__()) == Set("16")
map(ByteSet(16), lambda el: el ** 2) == IntSet(256)

So, if you want to build a type-preserving collections framework, you can look at Scala's collections framework and its builders and the CanBuildFrom implicit type-level machinery. I would suggest you look at the strawman proposal for the simplified and redesigned framework in Scala 2.13 instead of the current release since the latter one is considered to be overengineered and overly complex.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.