Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [pipelining]

The tag has no usage guidance.

0
votes
0answers
33 views

HTTP/REST and chained processing protocol/convention

Is there a protocol or a convention that supports REST (ok, maybe we should use HTTP here instead) processing chain and some neat features to help with that? Let me explain what I mean. Let's assume ...
0
votes
1answer
39 views

How could I pipeline two sequential workflows where there is room for overlap in the processes?

I'll break this post into two parts, because I'm trying to abstract the concept, but will explain my implementation at the end. I have two workflows, Workflow A and Workflow B. Part of Workflow B ...
0
votes
1answer
148 views

How to organise development process with AWS DataPipeline?

I'm introducing a Redshift for my team and it becomes a challenge to make a solid development process, so I need some info how to deal with it. First, of all some info about current layout. We are ...
0
votes
0answers
39 views

What principles describe data contiguity between systems?

I'm dealing with third-party data integration problems and need some theoretical principles, terms and topics for further research. So my question, What principles describe data contiguity between ...
3
votes
0answers
239 views

Perfect video processing pipeline?

I'm working on a modular video processing pipeline. It's currently presented as a tree of modules. Each module has a set of "results" and can dynamically request data results from other modules. Each ...
2
votes
0answers
175 views

Data pipeline with fallbacks and callbacks

I'm refactoring our current design for how we download static data. It's a mess of deep class hierarchy and callback hell and I want to convert it to a more elegant straightforward design. Here are ...
51
votes
3answers
3k views

What is the exact ingenuity of Unix pipe

I have heard the story of how Douglas Mcllroy came up with the concept and how Ken Thompson implemented it in one night. As far as I understand, pipe is a system call which shares a piece of memory ...
1
vote
1answer
139 views

pipeline step with two outputs which will be used by different later steps

I am creating a java package which offers an API based on a pipeline pattern. That is I have a series of steps which can be plugged together in any combination provided their inputs match the Output ...
5
votes
1answer
659 views

OO design in a data processing pipeline

I'm wondering how to design a fairly simple class whose properties are complex to compute. Also, the properties depend upon each other for computation. An example using graphs and graph processing (...
0
votes
1answer
123 views

Unix way of filtering/mapping/mapcatting records via the pipeline?

At home I use Linux and have for more than a decade. At work we use Windows and so I find PowerShell more familiar. I am a Linux/Unix fanboy at heart, but I've had greater success grokking the ...