1

I have a dataset full of rows that I must initialize into myclass and then process.

I am currently looping through each row in the dataset, initializing a new instance of myclass, then adding that instance to a list with type myclass.

I then loop through the list and conduct a process on each instance (for purpose of the example, let's say I must send each instance as a SOAP message).

My question is, do I keep the structure as is (with a list of myclass instances) or do I ditch the list and just process my instance (ie send the SOAP message) within the same loop that I initialize the object? I would basically leverage a new method within myclass to send this SOAP message.

I care about performance and memory usage. It is a batch process where I must process 20,0000 rows at a time. I have limited resources on the host server.

Additional info: If an initialization errors or a SOAP send fails, I want to continue with my processing of each row...

|improve this question
  • Either one could work. They have different directions of thought in the "get my head around what the developer is doing." – user40980 Feb 15 '16 at 16:57
  • 3
    The list approach has its advantages if you plan to do things like sorting the list, which may require the instances to be actual objects rather than the raw data from the DB. It is also useful in terms of splitting the code that retrieves the data and converts it from the code that uses it. But it probably consumes a little bit more resources (memory), though only actual testing will be able to tell. – jcaron Feb 15 '16 at 17:12
6

If you use a modern language like Python, C#, or Java, why not make use of iterators? Write one function which produces the objects and returns a lazy-evaluated iterator (in C# an IEnumerable<myclass>, in Java Iterable<myclass>), and another which iterates over the result of the first function and does the processing for each object.

That way, you can separate the concerns of object construction and processing without producing an intermediate list of objects in-memory. Moreover, I would not invest too much thoughts into any hypothetical performance penalties of this approach when one of the processing steps is "to send an instance as SOAP message over network"

Concerning initialization or processing errors: this is completely orthogonal to the discussed designs. You can handle this in one big loop, with an intermediate list or with an iterator, this does not make a big difference.

|improve this answer
  • I just wanted to report that this approach decreased my processing time by ~35% and used 30% less memory. Pretty awesome. – S1r-Lanzelot Feb 16 '16 at 17:28
  • 1
    @40Alpha: if it also increased the structure and the separation of concerns in your program, that would be a real benefit ;-) – Doc Brown Feb 16 '16 at 17:54
2

I think the performance difference would be negligible. It's just a matter of preference. The bottleneck would probably be the network anyway.

If you really want to be "economical" though, avoid building a huge list of objects altogether. Just iterate over your source data and reuse an object instance, or a pool of them if multi-threaded.

|improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.