Separation of data retrieval and processing in loops?

Question

Often I need to get some data and process it in some way. For example getting a list of customers from an API and assemble some summary data on them. As an example, getting :

api_result = api.request('customers/get')

for item in api_result:
    customer = Customer.fromDict(item)
    process_customer(customer)

This works but the data processing is coupled with the data retrieval. Ideally I'd like to be able to separate the retrieval of the customers from the processing so the code is less coupled. I'd like to instead be able to have something like:

def get_customers()

    customers = []
    api_result = api.request('customers/get')

    for item in api_result:
        customer = Customer.fromDict(item)
        customers.append(customer)
    
    return customers

...

customers = get_customers()

for customer in customers:
    process_customer(customer)
    

But this forces me to implement the loop twice. One loop is to hydrate the customer object from the API result, and then another loop is to process the data.

Is there a design pattern or way to separate these two steps yet still only loop over the data once? (Or is it just accepted that these will usually be two separate steps with two loops? I am using python, but I'm interested in the general case)

Answer 1

In general there is no shame in writing several loops over what seems to be the same data set. Loops are a fact of life and well-understood. If you dislike the verbosity, well, Python has list comprehensions and map to hide much of it.

In particular, looping over the output of an external system to create structures the way your program needs them to be is very common, and it's even to be recommended if there is the slightest chance that you will be doing more than one thing with these internal structures.

But often there will be other, more important constraints that dictate what to do. For instance, efficiency can force you to intermingle the steps more: if you first drain the external API and then do stuff with it, you have to keep its entire output in RAM at the same time, while if you stuff everything into one loop you can get away with much less memory. Having to rent a larger instance for a long-running micro-service costs a lot of money in the long run, so it is frequently the deciding factor.

Answer 2

You could optimize the loop in your example by adding a generator instead of the usual retrieval function:

def get_customers()

    api_result = api.request('customers/get')

    for item in api_result:
        customer = Customer.fromDict(item)
        yield customer

...

customers = get_customers()

for customer in customers:
    process_customer(customer)

Even if in the code there are 2 loops, you're only executing the loop within get_customers when necessary (when the processing function demands it).

This feature is available on other languages as well (e.g.: see yield return in C# -- https://stackoverflow.com/a/39496/6342009).