Questions tagged [stream-processing]
Stream processing is consuming an input stream or sequence of bytes with some format, processing the data as it arrives, and translating it into another format in a related output stream.
88 questions
1
vote
3
answers
120
views
Design of ETL system that should receive both files and via api
I am tasked to design a system that should receive data either as files or via an API and perform ETL functions. The end result is put into an RDBMS.
For the sake of example, imagine a system that ...
- 145
3
votes
4
answers
1k
views
Is real-time processing a synonym for streaming?
I watched two YouTube videos where in:
1st one - there are concepts of batch processing and stream ingestion and in the
2nd one - there is a comparison between batch and real time processing.
Is it ...
- 261
1
vote
2
answers
159
views
Serialization pattern and code duplication
I noticed how our codebase contains multiple versions of the same method, which unmarshalls the inbound byte-stream into java objects and that the only difference between the current and previous ...
- 141
-1
votes
1
answer
105
views
"Select All" in Table is now sending too much data to Server via API - Alternative Ways to Send to Server?
Outline of the current architecture of our web app outlining the issue I'm seeing
Client-side app is React, talking to a server running the Play! framework via an API.
On the page is a table that ...
- 99
0
votes
0
answers
61
views
How to achieve high performance routing to Kinesis streams?
I have consumer pools that consume only events matching a specific attribute (e.g. origin=WINDOWS, the other origin=LINUX).
Since ordering is important (ordering is based on instanceId, so this is the ...
user374484
2
votes
4
answers
821
views
How is processing done when streaming data?
If streams in programming is about reading data in chunks, how then is it possible to process the data in the scenarios where the data cannot be processed bit by bit, but the processing logic needs ...
- 171
2
votes
1
answer
782
views
Reprocessing out of order events in a stateful stream
Say that you have a monitoring application that reports downtimes for a large number of remote systems (think IOT). A monitoring daemon polls the remote system and reports that status (on/off) via an ...
- 746
1
vote
2
answers
238
views
API for input stream operation read until
I want to create a stream class The input stream should read/parse a continuous range from left to right providing convenience methods. The implementation isn't a problem but choosing a consistence ...
- 119
1
vote
1
answer
88
views
Showing posts by their time-decayed score
I want to show a number of posts, ordered by their score, limited to some number. The score is the product of the number of likes they have (from another table) and the recipricol of their normalized (...
- 630
0
votes
0
answers
50
views
Bigdata architecture: persisting raw data and streaming them at the same time
In a case where we have multiple source of data, we need to :
persist each source of data in its raw form.
process the data source eventually with each other to transform them in a presentable state (...
- 129
-1
votes
1
answer
77
views
Design for Maintaining changes over time which supports arbitrary updates in different time points
I have a collection of documents, which hold a subject id, a timestamp and a value. For example:
{ sid: 1, t: 3, v: "A" }
{ sid: 1, t: 5, v: "B" }
Which means subject#1 is ...
- 1,139
-3
votes
1
answer
345
views
How are different resolution settings for videos handled by server and client based on setting and connection speed? [closed]
I was watching youtube and squinted at the text in the video and wondered what happens when I switch from 720p to 1080p.
Does the streaming service, Youtube and or any other, just have the files in ...
0
votes
2
answers
73
views
Do 1:1 mobile video call face filters belong on Client or Server?
I am drafting architecture for 1:1 video calling mobile app with face filters (face recognition).
I could use p2p e.g. WebRTC because more savings less complexity. However, continuous face rec during ...
- 103
-5
votes
2
answers
240
views
IO stream interface in C
When writing a library (eg. for dealing with JPEG or PNG files), the first thing library author needs to provide is an abstraction over FILE* (to allow also reading from plain memory, pipe, socket, ...
- 99
0
votes
1
answer
60
views
System design to store pieces of data with specific ttyl to then refetch
I have a system which generates pieces of data that needs to be tracked and assigned a specific TTYL value (a value in seconds) for how long they are supposed to live before being re-processed.
I ...
- 109
0
votes
1
answer
406
views
How to keep a data warehouse updated?
Suppose there is a system ( like an ERP ) that writes to a database ( not too big, less than 100GB ). You need to export the data from this database to a data warehouse ( like RedShift or BigQuery ) ...
- 183
0
votes
1
answer
158
views
what is the correct OOP way to read/write binary data if meta information of how data is encoded is stored inside data itself?
I have a Buffer which wraps a stream of binary data.
The first byte indicates order, either big endian or little endian, which is needed to unpack the data.
class FooBuffer implements Buffer is
...
- 11
-1
votes
1
answer
52
views
Event Processing a Lack of Events
I am facing an issue that I don't believe is novel but none the less am having trouble finding a solution that fits well with our system. We have a constant stream of events going into AWS Kinesis. ...
- 107
3
votes
3
answers
3k
views
What is the right algorithm to match regex on a stream?
I'd like to match a regex pattern on a stream, but I am not sure what algorithm to use. I certainly don't want to load the entire file into memory.
I tried to figure out how to do this, but I have ...
- 1,259
0
votes
1
answer
91
views
Transitive matching in streaming application
Problem
A streaming application should perform matching transitively i.e. if A == B & B == C then A == C
Current Implementation
Application accepts domain objects in a streaming fashion and ...
- 111
-2
votes
1
answer
2k
views
Generate large Excel files and response from API
I need to generate a large Excel file (something around 50 megs) and send response to another API which will provide it to the front end for a download option.
My question is if it will be better to ...
- 389
2
votes
1
answer
233
views
Are streams of binary data considered a form of bit banging?
Are streams of binary data considered a form of bit banging? Does this definition change if the array is buffered? I am referring software which handles binary data on a general purpose CPU; for ...
- 191
1
vote
1
answer
784
views
What is the relationship between reactive programming and stream processing engines?
When would I use reactive programming libraries like RX Java and Project Reactor compared to stream processing engines such as Storm and Flink?
I am aware that these concepts might not be directly ...
- 157
1
vote
1
answer
423
views
How should I approach the comparison of two BMP images?
I am currently writing a Python program that retrieves the pixels from two BMP files and finds the percent difference between them (without using the Pillow library for the purpose of learning). I can ...
- 23
5
votes
2
answers
950
views
What are the distinction and relation between batch processing and stream processing systems?
Design Data Intensive Applications says
Batch processing systems (offline systems) Chapter 10
A batch processing system takes a large amount of input data, runs a
job to pro‐ cess it, and ...
- 5,495
0
votes
0
answers
341
views
What could be a good scalable alternative to polling data and then looping over a list of objects for triggering some action on those?
So I have some naive implementation for the following problem:
We have a list of objects with action methods that have to be trigger at some value of interest.
This value is polled (or is streamed) ...
- 101
1
vote
0
answers
137
views
Efficient Options to do Stream Joins(with Large Time gaps)
Use Case: I need to join two stream sources(say orders(order_id, order_val) and shipments(shipment_id, shipment_val)) based on an id(order_id = shipment_id) and generate a new event shipment_order(id, ...
- 129
2
votes
0
answers
418
views
Data pipeline architecture: airflow triggered by message broker
Let us say we have:
a web app with a Postgres DB that produces data over time,
another DB optimized for analytics that we would like to populate over time.
My goal is to build and monitor an ETL ...
- 151
0
votes
1
answer
349
views
Solving Event Dependence in Event Driven Systems
There are X Million Shipments per day. Each shipment has about 50 metrics to be calculated.
Each metric is calculated based on a type of the event(Let's say event_1 has the required information to ...
- 129
3
votes
0
answers
65
views
Moving application control flow from NiFi to code
We currently have an application that is nominally written in Java/Spring boot, but all of the control flow is in NiFi.
For example, there are the following layers in the java application:
...
- 303
3
votes
0
answers
499
views
Is the concept of Rx debounce possible in a distributed environment?
The issue
I am building a server-side system that handles streams of time-series events from multiple users and I'd like to perform an action after a certain quiet period has elapsed in the event ...
- 139
2
votes
1
answer
97
views
Aggregating events in event-driven system
Lets say there are two services. One generates event A and the other event B.
We need to build a new service that implements the function C = func(A, B) , which produces the result C.
But as A and B ...
- 129
0
votes
1
answer
165
views
Best practices for fixing bugs in event stream processing [closed]
We are working on an event driven system that works a streaming technology (eventhub/kinesis/kafka). Imagine some system is generating events that are sent to the event stream. Then there are ...
- 151
5
votes
1
answer
18k
views
Download file stream as it's being generated
Scenario
We are building a UI that allows users to query our data in bulk. The return format is CSV and there is a decent amount of processing that needs to happen, so this processing processing is ...
- 210
3
votes
1
answer
630
views
What is a good technique to process an event stream to ensure sequential or transactional consistency
I am trying to improve an event-driven processing system, which is having a few problems because the events are not guaranteed to arrive in the correct chronological sequence. This is due to batching ...
- 141
3
votes
2
answers
2k
views
Java 8 - When should I use java.util.Stream instead of java.util.Collection?
I started studying functional programming with JavaScript. After this, I started to study it with Java 8 (streams, lambdas and method reference) and I realised that I tend to use streams as much as ...
- 143
0
votes
1
answer
428
views
How to signal end of stream?
I am designing an interface for reading and writing video frames to various inputs and outputs. Stream operators seem to me a superb alternative to named functions for the task. This is the gist of it:...
- 7,149
1
vote
1
answer
156
views
Why use apache spark instead of just a normal worker?
Why should someone use a stream processing engine like Apache Spark, Flink, Hadoop instead of just a normal backend worker which works on something and returns the results as soon as it's done?
...
- 399
10
votes
3
answers
6k
views
Why do you want to avoid flushing stdout?
I stumbled upon a question in Codereview, and in one answer the feedback was to avoid std::endl because it flushes the stream. The full quote is:
I'd advise avoiding std::endl in general. Along with ...
- 1,438
-1
votes
1
answer
282
views
how simulate sequential consumers in Kafka
I have kafka consumer that get data from kafka and do some process if some circumstances meet and send it for another consumer to do it's job.
How can i build something like this?
One way I think ...
- 1,315
1
vote
1
answer
141
views
Backend architecture for a datastore with real-time events?
I need to build a data store for transactions that meets the following requirements:
A 'transaction' is effectively a state machine which moves through a number of statuses during its lifecycle and ...
- 111
0
votes
2
answers
286
views
Ideal Interprocess Communication
So, there is a workflow I feel needs to exist. It may already exist, and I don't know about it.
Consider the Linux command line workflow, the "motto" of Linux, more or less. You have a bunch of ...
- 145
-2
votes
1
answer
580
views
Short-circuiting an infinite Java 8 stream
Assume
You have some POJO Animal like
public class Animal { // some fields }
You have some enum AnimalType like
public enum AnimalType { // some animal types }
You have some method that given an ...
- 157
0
votes
1
answer
183
views
Best practices for reading dynamic byte streams: is line-by-line comparison with a mask the best way?
I receive sensor data as a binary stream of bytes. This stream is not always the same length, and does not include the same data set each time. If the sensor did not send a field, it is simply absent, ...
2
votes
2
answers
243
views
Receive data in random order, send in order
I am facing an issue where I have a data stream that sends unordered data. I'm trying to find a way to receive the data in random order, but send it in order.
As an example, I'll receive object4 and ...
- 167
6
votes
2
answers
122
views
Are there any techniques for detecting redundancies a stream of changes to a filesystem?
I'm working on a file-synchronization client that currently produces a stream of changes to the underlying filesystem. That is a stream of create/update/move/delete events is produced for each ...
- 2,198
0
votes
1
answer
841
views
Should I produce output with yield or stream.write in Python 3?
I am working on a Python 3 program that will among other things write data to text files in a number of formats. I have one function per format, generating the data to be written. I see two possible ...
- 1,361
-1
votes
1
answer
66
views
node server structure and technologies - support continuous and high volume (amount, not size) data transfare
I'm having trouble planning the structure of my server side workflow and the technologies I should use.
The basic structure and tasks are:
Now, things to consider:
1.the server listens to multiple "...
1
vote
1
answer
584
views
Why is the sorted() method on Stream not named sort()?
Why is the sorted() method on Stream not named sort()?
list.stream().sort()
- 29
2
votes
2
answers
2k
views
What are reactive streams and what's so special in them?
I browsed Wikipedia and encountered a new feature of Java 1.9: Reactive streams.
Unfortunately, the linked Wikipedia page doesn't help much to understand what those reactive streams are and what's so ...
- 2,579