Theoretical situation:
One trillion foobars are stored in a text file (no fancy databases). Each foobar must have some business logic executed on it. A set of 1 trillion will not fit in memory so the data layer cannot return a big set to the business layer. Instead they are to be streamed in 1 foobar at a time, and have business logic execute on 1 foobar at a time. The stream must be closed when finished.

In order for the stream to be closed the business layer must close the stream (a data operation detail), thus violating the separation of concerns.

Is it possible to incrementally process data without violating layers?

  • Why you should process all records at one open stream? – Yusubov Oct 5 '12 at 2:05
  • @ElYusubov. Incremental processing of any nature runs into the same issue reguardless of 1 to N streams. – Lord Tydus Oct 6 '12 at 0:46

Good question.

Here is a C#-ish code that lets you have it both ways. The trick here is lazy evaluation (yield).

// Lazy producer that will auto-close the file.
// The OS and the disk do the caching for you.
public static IEnumerable<FooBar> GetFoobars(string fileName, long maxNumber = 1000000000000)
    using(File file = open(fileName)) // pseudocode
        FooBar nextFooBar;
            nextFoobar = new Foobar(file.ReadByte()); // or something like that.
            yield return nextFoobar;
        while(nextFooBar != null || /* reach max */); // pseudocode

// Consumer ... stupid example. Oh well ...
public static int Consume()
    int result = (from foo in GetFoobars("foo.txt").AsParallel()
                 where foo.Depth % 10 == 0).Count();
    return result;

The GetFoobars(...) function know which file to open, how many thingies to read and how to close the stream. You just pull on the results as you need them. Concerns are separated. What seems to be the problem? You could probably do the whole thing as a one-liner ... maybe not, but close.

P.S. I recommend going through SICP video lectures.

  • Accepted for "yield return". – Lord Tydus Oct 6 '12 at 0:45
  • 1
    Note that technically yield return does not have to be present in the language, but boy does it make life easier when it is! startbigthinksmall.wordpress.com/2008/06/09/… – Job Oct 7 '12 at 2:55
  • 1
    Surely it is not the yield return that answers your question; it’s the IEnumerable<T> interface and the semantics that it abstracts. The data layer can implement IEnumerable<T> without the use of yield return and the business layer wouldn’t have to care about that. – Timwi Oct 7 '12 at 6:22
  • 1
    @LordTydus: No. See my answer. The data layer closes the stream when the enumeration is either completed or aborted. The business layer doesn’t know and doesn’t care that this happens. (This should have been obvious to you because yield return is just syntactic sugar. It is possible to write equivalent code without using yield return.) – Timwi Oct 8 '12 at 6:59
  • 1
    That's exactly what I'd have done in Java. Unfortunately Java does not have yield return, but it does have the Iterable and Iterator interface, which allow the same effect to be achieved (with slightly more convoluted code, however). – Joachim Sauer Oct 8 '12 at 7:22

Is it possible to incrementally process data without violating layers?

Yes, it is. You have to define unit of work which will solve your confusion. I think processing set of records (100 or 1000 depend on processing time of an average record) or a record at a time will have more reliable and manageable results.

In addition, i would make sure that data manipulation process (business processing) is in a transaction, all error processing's are recorded and rolled back for investigation. You may always have some exceptional situations which must be taken into account.


Define an abstract interface that presents a foobar (or whatever set of foobars is required), leaving it up to the implementation of the abstraction to determine how the foobars are acquired and/or modified.

Naturally, you need to define the elements of this interface around that actual conditions. If the only way to count the foobars is to stream past all trillion, don't include "get the number of foobars" in your interface. But probably your interface will contain some elements like "start" "stop" and "abort" in addition to "get"


The way I see it is this:

  • You have a data layer which provides functionality to read individual foobars, one at a time.

  • This functionality has an API. That API has a contract. Anyone wishing to use the API must adhere to the contract. The contract might specify, for example, that you must call NextFoobar() until you get null or something, and then you must stop calling it, but of course it could specify something more complex and it would simply be the contract to use.

  • The business layer doesn’t care how you implemented the API, it only needs to use this API to do what it needs to do. Using the API includes that it must adhere to the contract.

Now, if you look at the C#-based answer posted by @Job, it follows all of these principles:

  • The data layer provides an IEnumerable<Foobar> — an object that allows anyone to retrieve foobars from it, one at a time, without having to care where the foobars are coming from.

  • The API on IEnumerable<T> is well-defined. You can either use it the hard way (call GetEnumerator(), and then on the enumerator keep calling MoveNext() and Current until MoveNext() returns false, and then call Dispose()) or use a language feature or method that uses the API automatically (e.g. foreach or, in his case, .Count()).

  • The business layer correctly uses this API. It does need to call Dispose() (which it does indirectly by calling .Count()) but it doesn’t need to care what Dispose() actually does, it only calls it to fulfill the contract.

(Incidentally, if you enumerate all foobars, the file is not actually closed by the call to Dispose() but by the last call to MoveNext()... but Dispose() is still part of the API so that it can (and does) close the file if you decide to stop enumerating in the middle. The beauty of separation-of-concerns is that the business layer doesn’t need to care about any of that.)

  • +1 Certainly a deeper answer than mine. This question would have had an answer 10 years ago; the only difference is that now tools are better in general and more patterns have made their way into languages to become a syntactic sugar. LINQ is one of them. – Job Oct 7 '12 at 22:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.