I'm working on a modular video processing pipeline. It's currently presented as a tree of modules. Each module has a set of "results" and can dynamically request data results from other modules. Each request and result are marked with frame ID. User requests one or more "metrics", i.e. end results, for a set of frames (each frame, or selected frames).

For example, the top of the tree is the video reader. Then, there is a module of scene edge detection, which requires frames i-1 and i from video reader to generate results for frame i. There are modules for various information about video frames: histograms, edge maps, occlusion maps, disparity maps, etc. Many results are shared between modules. Some modules require data from a single frame, some—from a fixed number of consecutive frames; some, given request for a frame, start "exploring" data from neighboring frames in both directions—nobody knows in advance, how much data they will need.

Data requests are handled by a centralized service, which owns instantiations of all modules and knows, what module produces what data. It has the right to cache data for further usage by other modules or output it (in the case of metrics).

It all works well in theory, but in practice, I met some problems. How long should the cache hold the data portions? If I don't cache at all, then the same frame will be read multiple times. Some "exploring" metric (see above) may need it later. If I cache everything, then in the worst case the whole video (or its large portion) will be loaded into memory—which is unacceptable.

This cache is the problem my question is about. I have full control over the sources. Modules can tell the caching service any information they want about themselves.

The perfect solution would never have to generate the same data twice and would keep memory usage at a minimum (get rid of data as soon as possible). Well, I don't expect to hear this perfect solution, but some patterns, which would help me come as close to it as possible.

  • 1
    While targeted at a lower level (e.g. video codecs), the techniques behind StreamIt may be useful. StreamIt is primarily a synchronous data flow language, but it supports having dynamically scheduled streams as well. Mar 4, 2017 at 18:44
  • @DerekElkins I can easily see how the data flow can be optimized—if not for those modules, which don't know how many frames they will need to observe. I couldn't see any hint how StreamIt deals with those.
    – Anton3
    Mar 5, 2017 at 11:21
  • 1
    How much research have you done in investigating existing pipelines such as DirectShow, and what elements of their designs do you find unsuitable? May 18, 2017 at 16:37

1 Answer 1


Take a look at the Imaging Whiteboard. This is a collection of 43 modules which are 'drawn' on the whiteboard and connected as desired. Processing is near real-time so images are only held in memory as required, input is from camera or video file. Output to monitor or video file.

Not the answer you're looking for? Browse other questions tagged or ask your own question.