8

Usually I achieve low coupling by creating classes that exchange lists, sets, and maps between them. Now I am developing a Java batch application and I can't put all the data inside a data structure because there isn't enough memory. I have to read and process one chunk of data and then going to the next one. So having low coupling is much more difficult because I have to check somewhere if there is still data to read, etc.

What I am using now is:

Source -> Process -> Persist

The classes that process have to ask to the Source classes if there are more rows to read.

What are the best practices and or useful patterns in such situations?

I hope I am explaining myself, if not tell me.

| improve this question | |
  • 3
    one of the ways to achieve low coupling is to establish a good protocol of communication between source classes and process classes – treecoder Apr 9 '12 at 9:19
  • 3
    I think you might want to look at using a message queue - some kind of data bus - so that your classes put things into queues and pull them from queues rather than interacting directly. – Murph Apr 9 '12 at 9:49
  • @Murph is there a simple way or a good Java library to use a message queue? – Виталий Олегович Apr 9 '12 at 9:57
  • @vitalik - I'm a .NET developer and still feeling my way with message queues generally so not really in a position to provide a confident answer (hence my response being to comment) – Murph Apr 9 '12 at 10:09
  • 1
    @Murph ok, thank you anyway! I guess I will start studying queues too! – Виталий Олегович Apr 9 '12 at 13:04
7

From the comments I see you're using Java. Have a look at various Queue implementations. Particularly, BlockingQueue is useful for producer-consumer scenarios. You could have two queues: one between Source (producer of data) and Process (consumer of data), and another between Process (producer of results) and Persist (consumer of results).

With limited-capacity blocking queues it's fairly easy to implement efficient (the bottleneck part, whatever it is, is kept fed with data 100% of the time) systems, still using only a bound amount of memory no matter how much data there is.

| improve this answer | |
  • Your solution is very good. But what happends if I use a limited capacity queue, and the queue is full and I try to add something to it? – Виталий Олегович Apr 9 '12 at 12:21
  • @vitalik then you have to put a strategy in place such as temporarily storing the data in an in memory DB or to disk other other such solution. – Martijn Verburg Apr 9 '12 at 12:50
  • @MartijnVerburg yes, but I guess it would be easier if there was a possibility to make to sleep the producer until there's more space available in the queue. – Виталий Олегович Apr 9 '12 at 13:21
  • 1
    @vitalik of course there is that possibility (to sleep a producer) you just have to do it. Some queues can be configured to be blocking, so that if a producer tries to insert into a full queue, you just block, and effectively sleep/spin (watch out for which one) on the queue to have space. – sdg Apr 9 '12 at 14:51
  • 1
    @vitalik: See e.g. BlockingQueue.put docs: Inserts the specified element into this queue, waiting if necessary for space to become available. Simple and convenient!:) – Joonas Pulakka Apr 9 '12 at 16:45
2

A blocking queue (from Joonas Pulakka) is the heavy-duty answer. A simpler answer might work. If you have all the data stored in the source, you can just pass a reference to the processor, and it can just grab the data out of the source. Of course, this is probably what you were doing in the past. You may not have all the data in memory in the source and you may not get the low coupling you want.

The next step up would be to use an Enumerator or Iterator interface. (Iterators are more common in Java, though most times that remove method is just a nusance.) The processor would obtain the Iterator from the source, then call the methods until done. If the source is pulling terrabytes of data from somewhere, each call might take a while. But if you're going to sleep the processor until there's something in the queue anyway, this will just be doing that automatically. And if the source gets ahead of the producer, the source will automatically wait for the producer to call hasNext and next.

If, on the other hand, you want the source grabbing data from its source as fast as it can and stockpiling it until the processor catches up, not sitting around waiting for the processor to process, then the queue--and multiple threads--start to look like a good, if more complicated, idea. Now the source can pile up the data when it can run faster (its limit presumably being something like disk I/O), and the processor can reduce the size of the pile when it can run faster, (its limit being how fast the persistance module can persist the data).

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.