A blocking queue (from Joonas Pulakka) is the heavy-duty answer. A simpler answer might work. If you have all the data stored in the source, you can just pass a reference to the processor, and it can just grab the data out of the source. Of course, this is probably what you were doing in the past. You may not have all the data in memory in the source and you may not get the low coupling you want.
The next step up would be to use an Enumerator or Iterator interface. (Iterators are more common in Java, though most times that remove
method is just a nusance.) The processor would obtain the Iterator from the source, then call the methods until done. If the source is pulling terrabytes of data from somewhere, each call might take a while. But if you're going to sleep the processor until there's something in the queue anyway, this will just be doing that automatically. And if the source gets ahead of the producer, the source will automatically wait for the producer to call hasNext
and next
.
If, on the other hand, you want the source grabbing data from its source as fast as it can and stockpiling it until the processor catches up, not sitting around waiting for the processor to process, then the queue--and multiple threads--start to look like a good, if more complicated, idea. Now the source can pile up the data when it can run faster (its limit presumably being something like disk I/O), and the processor can reduce the size of the pile when it can run faster, (its limit being how fast the persistance module can persist the data).