Questions tagged [stream-processing]

Stream processing is consuming an input stream or sequence of bytes with some format, processing the data as it arrives, and translating it into another format in a related output stream.

Filter by
Sorted by
Tagged with
0
votes
1answer
26 views

How to keep a data warehouse updated?

Suppose there is a system ( like an ERP ) that writes to a database ( not too big, less than 100GB ). You need to export the data from this database to a data warehouse ( like RedShift or BigQuery ) ...
0
votes
1answer
87 views

what is the correct OOP way to read/write binary data if meta information of how data is encoded is stored inside data itself?

I have a Buffer which wraps a stream of binary data. The first byte indicates order, either big endian or little endian, which is needed to unpack the data. class FooBuffer implements Buffer is ...
-1
votes
1answer
32 views

Event Processing a Lack of Events

I am facing an issue that I don't believe is novel but none the less am having trouble finding a solution that fits well with our system. We have a constant stream of events going into AWS Kinesis. ...
1
vote
2answers
193 views

What is the right algorithm to match regex on a stream?

I'd like to match a regex pattern on a stream, but I am not sure what algorithm to use. I certainly don't want to load the entire file into memory. I tried to figure out how to do this, but I have ...
0
votes
1answer
59 views

Transitive matching in streaming application

Problem A streaming application should perform matching transitively i.e. if A == B & B == C then A == C Current Implementation Application accepts domain objects in a streaming fashion and ...
-2
votes
1answer
48 views

Generate large Excel files and response from API

I need to generate a large Excel file (something around 50 megs) and send response to another API which will provide it to the front end for a download option. My question is if it will be better to ...
2
votes
1answer
211 views

Are streams of binary data considered a form of bit banging?

Are streams of binary data considered a form of bit banging? Does this definition change if the array is buffered? I am referring software which handles binary data on a general purpose CPU; for ...
0
votes
1answer
77 views

What is the relationship between reactive programming and stream processing engines?

When would I use reactive programming libraries like RX Java and Project Reactor compared to stream processing engines such as Storm and Flink? I am aware that these concepts might not be directly ...
1
vote
1answer
79 views

How should I approach the comparison of two BMP images?

I am currently writing a Python program that retrieves the pixels from two BMP files and finds the percent difference between them (without using the Pillow library for the purpose of learning). I can ...
4
votes
2answers
103 views

What are the distinction and relation between batch processing and stream processing systems?

Design Data Intensive Applications says Batch processing systems (offline systems) Chapter 10 A batch processing system takes a large amount of input data, runs a job to pro‐ cess it, and ...
0
votes
0answers
56 views

RS232 3rd party performance using C#

In my work i have an old fashion Vb app.. It's take single sale line(by text change event..) from the input and send it through RS232 output native. In purpose to get rid from any led display i took a ...
0
votes
0answers
69 views

What could be a good scalable alternative to polling data and then looping over a list of objects for triggering some action on those?

So I have some naive implementation for the following problem: We have a list of objects with action methods that have to be trigger at some value of interest. This value is polled (or is streamed) ...
0
votes
0answers
32 views

Efficient Options to do Stream Joins(with Large Time gaps)

Use Case: I need to join two stream sources(say orders(order_id, order_val) and shipments(shipment_id, shipment_val)) based on an id(order_id = shipment_id) and generate a new event shipment_order(id, ...
2
votes
0answers
146 views

Data pipeline architecture: airflow triggered by message broker

Let us say we have: a web app with a Postgres DB that produces data over time, another DB optimized for analytics that we would like to populate over time. My goal is to build and monitor an ETL ...
0
votes
1answer
93 views

Solving Event Dependence in Event Driven Systems

There are 60 Million Shipments per day. Each shipment has about 50 metrics to be calculated. Each metric is calculated based on a type of the event(Let's say event_1 has the required information to ...
3
votes
0answers
32 views

Moving application control flow from NiFi to code

We currently have an application that is nominally written in Java/Spring boot, but all of the control flow is in NiFi. For example, there are the following layers in the java application: ...
2
votes
0answers
130 views

Is the concept of Rx debounce possible in a distributed environment?

The issue I am building a server-side system that handles streams of time-series events from multiple users and I'd like to perform an action after a certain quiet period has elapsed in the event ...
1
vote
1answer
60 views

Aggregating events in event-driven system

Lets say there are two services. One generates event A and the other event B. We need to build a new service that implements the function C = func(A, B) , which produces the result C. But as A and B ...
0
votes
1answer
37 views

Best practices for fixing bugs in event stream processing [closed]

We are working on an event driven system that works a streaming technology (eventhub/kinesis/kafka). Imagine some system is generating events that are sent to the event stream. Then there are ...
1
vote
1answer
4k views

Download file stream as it's being generated

Scenario We are building a UI that allows users to query our data in bulk. The return format is CSV and there is a decent amount of processing that needs to happen, so this processing processing is ...
2
votes
1answer
119 views

What is a good technique to process an event stream to ensure sequential or transactional consistency

I am trying to improve an event-driven processing system, which is having a few problems because the events are not guaranteed to arrive in the correct chronological sequence. This is due to batching ...
2
votes
2answers
616 views

Java 8 - When should I use java.util.Stream instead of java.util.Collection?

I started studying functional programming with JavaScript. After this, I started to study it with Java 8 (streams, lambdas and method reference) and I realised that I tend to use streams as much as ...
0
votes
1answer
152 views

How to signal end of stream?

I am designing an interface for reading and writing video frames to various inputs and outputs. Stream operators seem to me a superb alternative to named functions for the task. This is the gist of it:...
0
votes
1answer
80 views

Why use apache spark instead of just a normal worker?

Why should someone use a stream processing engine like Apache Spark, Flink, Hadoop instead of just a normal backend worker which works on something and returns the results as soon as it's done? ...
6
votes
3answers
1k views

Why do you want to avoid flushing stdout?

I stumbled upon a question in Codereview, and in one answer the feedback was to avoid std::endl because it flushes the stream. The full quote is: I'd advise avoiding std::endl in general. Along with ...
1
vote
1answer
61 views

how simulate sequential consumers in Kafka

I have kafka consumer that get data from kafka and do some process if some circumstances meet and send it for another consumer to do it's job. How can i build something like this? One way I think ...
1
vote
1answer
94 views

Backend architecture for a datastore with real-time events?

I need to build a data store for transactions that meets the following requirements: A 'transaction' is effectively a state machine which moves through a number of statuses during its lifecycle and ...
0
votes
2answers
175 views

Ideal Interprocess Communication

So, there is a workflow I feel needs to exist. It may already exist, and I don't know about it. Consider the Linux command line workflow, the "motto" of Linux, more or less. You have a bunch of ...
-2
votes
1answer
549 views

Short-circuiting an infinite Java 8 stream

Assume You have some POJO Animal like public class Animal { // some fields } You have some enum AnimalType like public enum AnimalType { // some animal types } You have some method that given an ...
0
votes
1answer
50 views

Best practices for reading dynamic byte streams: is line-by-line comparison with a mask the best way?

I receive sensor data as a binary stream of bytes. This stream is not always the same length, and does not include the same data set each time. If the sensor did not send a field, it is simply absent, ...
2
votes
2answers
150 views

Receive data in random order, send in order

I am facing an issue where I have a data stream that sends unordered data. I'm trying to find a way to receive the data in random order, but send it in order. As an example, I'll receive object4 and ...
6
votes
2answers
107 views

Are there any techniques for detecting redundancies a stream of changes to a filesystem?

I'm working on a file-synchronization client that currently produces a stream of changes to the underlying filesystem. That is a stream of create/update/move/delete events is produced for each ...
0
votes
1answer
518 views

Should I produce output with yield or stream.write in Python 3?

I am working on a Python 3 program that will among other things write data to text files in a number of formats. I have one function per format, generating the data to be written. I see two possible ...
-1
votes
1answer
54 views

node server structure and technologies - support continuous and high volume (amount, not size) data transfare

I'm having trouble planning the structure of my server side workflow and the technologies I should use. The basic structure and tasks are: Now, things to consider: 1.the server listens to multiple "...
1
vote
1answer
218 views

Why is the sorted() method on Stream not named sort()?

Why is the sorted() method on Stream not named sort()? list.stream().sort()
1
vote
2answers
2k views

What are reactive streams and what's so special in them?

I browsed Wikipedia and encountered a new feature of Java 1.9: Reactive streams. Unfortunately, the linked Wikipedia page doesn't help much to understand what those reactive streams are and what's so ...
4
votes
2answers
372 views

Are .NET Streams like 'real' streams?

To me a stream implies that I should be able to: Put things in one end and receive it from the other end in the same order. Do these things at the same time. i.e. continuously be adding to the top ...
0
votes
1answer
3k views

Usage of For-each loop vs functional operation vs Java8 streams

What are the gains in actual computing speed (verbosity of code being put aside) and for which case would each be recommended for, as a good design pattern? I would like to know the above in general ...
14
votes
2answers
7k views

Traditional Message Brokers and Streaming Data

According to the Kafka site: "Kakfa is used for building real-time data pipelines and streaming apps." Searching the internet far and wide, I've found the following generally-accepted definition ...
1
vote
4answers
912 views

What is a “streaming app”, exactly?

I'm trying to get past all the hipster, pie-in-the-sky buzzwording and address a very simple, fundamental question: What is a streaming application? According to the Kafka site "Kafka is used for ...
1
vote
1answer
184 views

Moving from Batch to Streaming architecture using a Graph datastructure

The use case I am trying to solve is, assigning millions of users to their groups/segments. I have thousands of different criteria from which buckets of users are created. E.g. bucket criteria: All ...
4
votes
2answers
768 views

Differences between streams and iterators in C++?

I'm working on code which will provide a generic C++ interface to relational datasets. Some of the data will be in tables in memory (e.g. std::set or std::vector), but some of it will be computed - ...
2
votes
5answers
233 views

Algorithm to limit the speed of processing items

I need to process a large number of data items, and I need to restrict the speed at which they are processed. For example, not more than 20 items per minute. I've thought about an algorithm for that, ...
1
vote
0answers
610 views

Streaming architecture (aka Kappa) and different data types

I'm trying to check my understanding of the kappa architecture. The main point I am trying to resolve is how, if at all, one unifies the processing of data under analysis and other pieces of data, ...
1
vote
0answers
578 views

Async stream enrichment with a message queue (e.g. Kafka)

I would like to enrich some streaming data, with one service per enrichment, possibly each service written in a different language and managed/deployed separately. Each service would consume some ...
4
votes
3answers
175 views

Could one swap the direction of >> and << overloads to define them as class members?

Some operators cannot be overloaded as class members. One such example are the bitwise shift operators used for streams see here for example. The reason for that (as far as I understand it) seems to ...
1
vote
2answers
2k views

Video streaming through Ajax calls

These days I have seen couple of sites (including Youtube) that stream videos through Ajax calls (XHR) in few MBs or even less than MB. For an example of what I said see the highlighted lines in the ...
0
votes
1answer
116 views

Where should I store stream wide metadata in stream processing?

I am building a stream processing architecture and was wondering what to do with metadata about the stream. For instance, every message of data coming from a source has the same attribution, as it ...
10
votes
1answer
3k views

How does sorting with java 8 stream work under the hood?

When I call Stream.sort(..) is there a new array of elements created and the stream iterates over the newly created sorted array? In other words, how Java 8 Stream does sort under the hood?
6
votes
1answer
555 views

Why does the C stdio 'ungetc' function exist?

In the C programming language (and many subsequent languages that either directly interfaced with or built a facsimile of the C's Standard IO functions), there exists a function called ungetc: int ...