Questions tagged [pipelining]
The pipelining tag has no usage guidance.
17
questions
0
votes
2
answers
155
views
Does Automated Pipeline Mean CI/CD Pipeline?
I am not sure if I have been using the term wrongly (and including it in my CV), so some inputs from the community will be appreciated.
I am not a DevOps, but a noob machine learning engineer. So I ...
- 121
1
vote
1
answer
144
views
Data pipeline design - robust and resilient to future variations
I need to build a data pipeline to populate a database from various files. This is a common scenario. However, I want to have expert opinions for implementing a pipeline that is robust, modular and ...
- 23
1
vote
0
answers
129
views
Design patterns for versioning steps across data/workflow pipelines?
I'm sure this has been touched upon by a number of questions, but I'm struggling on drawing the boundaries between code, data and configuration versions when working with a large DAG (think airflow or ...
- 343
-3
votes
1
answer
610
views
Should the Jira assignee change for testing? [closed]
We currently have Reporter and Assignee fields for Jira stories. For example, let's say we move from the 'Code review' column to the 'Ready for testing' column. Should the tester then come in, delete ...
- 111
1
vote
1
answer
571
views
Should I use different Buildspec files for branch builds and deployment builds?
Very recently a discussion came up regarding the usage of different buildspec.yaml files, one for branch builds and the other for deployment builds and I was wondering, since after some research I ...
- 113
1
vote
3
answers
527
views
What makes a data pipeline scalable? Best practices for scalable design?
I have been searching about this topic for a few days and have not yet found anything on books, courses or tutorials.
What is a way to make data pipelines more scalable, that doesn't involve NoSql or ...
- 99
0
votes
3
answers
505
views
Is microservices architecture a good candidate for a pipeline?
I have a monolithic application which can be divided into multiple steps and different steps have variable scaling requirement, however the input of the next step is the output of the current step. ...
- 269
1
vote
2
answers
500
views
Remove all side-effects from business logic
I'm looking for feedback for a design pattern that aims to remove all side-effects from business logic. I'm using PHP but the pattern can be applied to any OOP language. The point is to enforce pure ...
- 145
5
votes
2
answers
937
views
What are the distinction and relation between batch processing and stream processing systems?
Design Data Intensive Applications says
Batch processing systems (offline systems) Chapter 10
A batch processing system takes a large amount of input data, runs a
job to pro‐ cess it, and ...
- 5,495
1
vote
0
answers
51
views
HTTP/REST and chained processing protocol/convention
Is there a protocol or a convention that supports REST (ok, maybe we should use HTTP here instead) processing chain and some neat features to help with that? Let me explain what I mean.
Let's assume ...
0
votes
1
answer
64
views
How could I pipeline two sequential workflows where there is room for overlap in the processes?
I'll break this post into two parts, because I'm trying to abstract the concept, but will explain my implementation at the end.
I have two workflows, Workflow A and Workflow B. Part of Workflow B ...
- 181
1
vote
1
answer
509
views
Perfect video processing pipeline?
I'm working on a modular video processing pipeline. It's currently presented as a tree of modules. Each module has a set of "results" and can dynamically request data results from other modules. Each ...
- 127
2
votes
0
answers
249
views
Data pipeline with fallbacks and callbacks
I'm refactoring our current design for how we download static data. It's a mess of deep class hierarchy and callback hell and I want to convert it to a more elegant straightforward design.
Here are ...
- 121
54
votes
5
answers
4k
views
What is the exact ingenuity of Unix pipe
I have heard the story of how Douglas Mcllroy came up with the concept and how Ken Thompson implemented it in one night.
As far as I understand, pipe is a system call which shares a piece of memory ...
- 671
1
vote
1
answer
838
views
pipeline step with two outputs which will be used by different later steps
I am creating a java package which offers an API based on a pipeline pattern. That is I have a series of steps which can be plugged together in any combination provided their inputs match the Output ...
- 183
5
votes
1
answer
1k
views
OO design in a data processing pipeline
I'm wondering how to design a fairly simple class whose properties are complex to compute. Also, the properties depend upon each other for computation.
An example using graphs and graph processing (...
- 1,447
0
votes
1
answer
204
views
Unix way of filtering/mapping/mapcatting records via the pipeline?
At home I use Linux and have for more than a decade. At work we use Windows and so I find PowerShell more familiar. I am a Linux/Unix fanboy at heart, but I've had greater success grokking the ...
- 1,718