# Using higher order functions to apply m out of M filter's and then transform data of size n?

Total no: of filters possible is M . User can select m filters where m <= M . A typical example is files from a folder , he could say modified between so and so date , start with so and so and so on.

``````meets = []
for each in something:
if user_filter1:
if user_filter1_condition_met:
meets.append(True)
#hardcode all other filters here
if all(meets):
do_something(each)

else:
meets = True
for each in something:
do_something(each)
``````

Here I am repeating the entire code again which is against DRY principles.

``````for each in something:
if user_filter:
if not filter_condition_met:
continue
do_something(each)
``````

Here I am doing the filter check for all items in the loop, while essentially it needs to be done just once.

Can I use higher order functions to check the filter condition once and generate a code that does the job? It looks like a filter and then transform operation. Can someone give some ideas?

An addendum to the question with a similar essence:

``````def function(inner,outer):
for each in outer:
for each in inner:
print "Something"

#Border condition inner doesn't exist I need a code object that goes like this:

def function(inner,outer):
for each in outer:
print "Something"

#Inner doesn't exist , so I want to to not go a for each in inner operation i.
#I mean it won't change the complexity of the problem at hand
#But suppose there are billions elements , I save billion for each in [].
``````
• Can you make your code snippets better follow your updated description? You're not looping over the filters in them and it makes your question difficult to follow. Dec 30 '14 at 21:57
• Really sorry. Does the above snippet explain the problem? Sorry for the terrible code because I am trying to solve it! I am learning higher order functions things like map reduce etc. Dec 30 '14 at 22:06
• What's your goal here? Why don't you want to check the filter in a loop? Why don't you want to iterate over the items twice? The code sample you've given can certainly be improved, but its not really case I'd typically go for higher order functions. Dec 30 '14 at 22:19
• Python has the concept of generators doesn't it? I would be surprised if there wasn't something akin to .NET style LINQ filtering... Dec 30 '14 at 22:22
• Answered my question using some help from freenode in one way. @WinstonEwert: I am sorry I had some time communicating my requirement to you. Please let me know if you find a different way to solve it. I am just trying to see ways to solve more than a single solution especially using higher order functions, I feel there could be higher paradigms to solve this. Dec 30 '14 at 22:32

Conceptually, each filter is a predicate - a function that given an input, returns either true or false. The python function `filter(function, iterable)` accepts a predicate and an iterable, and returns only items that the predicate is true for. So what you want is a way to combine all the filters a user selects into one filter, that only matches if all of them match. The `all` function can be used here, `all(iterable)` returns true only if every element of `iterable` is true. Using this, we can combine predicates:

``````def andPredicate(predicates):
def pred(v):
return all((p(v) for p in predicates))
return pred
``````

The `andPredicates` function accepts a list of predicates and gives you a new one that only returns true for an item if all the predicates return true for that item. For example, if you have predicates `isNumber` and `isLessThanZero`, you can define `isNegativeNumber` like this:

``````isNegativeNumber = andPredicate([isNumber, isLessThanZero])
print(isNegativeNumber("foo")) # False
print(isNegativeNumber(8)) # False
print(isNegativeNumber(-5)) # True
``````

So this lets you combine filters. If you have a list of filters, where each filter is a predicate, then you can just do this:

``````filteredItems = filter(andPredicate(userFilters), items)
``````

Done and done. The iterable is only iterated once, and the filters are checked one at a time on an item until one fails. If one fails, the rest of the filters are skipped and the next item is taken, if they all succeed, then the item is included in the final iterable. Note that `filter` produces a list in Python 2.x and an iterator in Python 3.x.

Additionally, the definition of `andPredicates` can be further shortened with an anonymous function, but that may be unclear to readers of your code if they're not familiar with functional programming patterns:

``````def andPredicate(predicates):
return lambda v: all((p(v) for p in predicates))
``````

For completeness, here are definitions of the aforementioned `isNumber` and `isLessThanZero` predicates:

``````from numbers import Number

def isNumber(v):
return isinstance(v, Number)

def isLessThanZero(x):
return x < 0
``````

In the second part of your question, you want to either do something once for each item in the cross product of two lists, or you want to do something once for each item in the second list of the first list is empty. The `itertools` module contains a function `product` that basically does the same thing as nested for loops. For instance instead of this:

``````for a in as:
for b in bs:
print("Something!")
``````

You can write this:

``````from itertools import product

for a, b in product(as, bs):
print("Something!")
``````

Both pieces of code are equivalent. When one of the iterables is empty however, the whole product is empty, just like in nested for loops. We can write our own version of `product` that instead of being empty when one of the iterables is empty, is only empty when both are empty and if just one is empty, then it just returns the values in the other iterable and uses `None` as the values for the first iterable. This can be defined like so:

``````from itertools import product

def productWithNones(as, bs):
if as and bs:
return product(as, bs)
if as:
return product(as, [None])
if bs:
return product([None], bs)
return []
``````

Now this:

``````for a, b in productWithNones([1, 2, 3], []):
print((a, b))
``````

Prints this:

``````(1, None)
(2, None)
(3, None)
``````

Instead of simply being empty. So your original loop can be written as:

``````for eachOuter, eachInner in productWithNones(outer, inner):
print("Something!")
``````
• Right you are composing filters which is handy sometimes especially the examples you gave. If possible please have a look at the addendum question. Dec 31 '14 at 10:28
– Jack
Dec 31 '14 at 10:37
• Yes the second one is fairly straight forward, I am trying to move it into a function to avoid the repetition! Dec 31 '14 at 10:38
• Ah, so you want the iteration logic separated from what the loop is actually doing?
– Jack
Dec 31 '14 at 10:39
• Functional programming can be quite efficient, but it's real benefit is clarity and expressiveness :) Editing with a new approach right now
– Jack
Dec 31 '14 at 10:43

Here I am doing the filter check for all items in the loop, while essentially it needs to be done just once.

Why is that a problem?

Write code in the cleanest way possible. If it makes sense to structure the code with the boolean test made multiple times, then write it that way.

Avoid unneeded optimizations, to quote Knuth, "premature optimization is the root of all evil".

• A very good perspective. I am just trying to learn and this won't hit production any time soon =) Dec 30 '14 at 22:16

Something like this is what I wanted: (Applying a sample use case I could come up with)

``````predicates = []

if filenamme:
predicates.append(lambda x,filename: x.name == filename)
if from_dt and to_dt:
predicates.append(lambda x, from_dt, to_date: from_dt <= x.mod_dt <= to_dt)

for file in files:
if all(predicate(file) for each in predicates):
do_something(file)
``````

I wanted to generate my predicates outside.

• This code won't actually work. Your lambda take multiple parameters, but you only pass in one when using it. But the basic approach should work. Dec 30 '14 at 22:35