Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [stream-processing]

Stream processing is consuming an input stream or sequence of bytes with some format, processing the data as it arrives, and translating it into another format in a related output stream.

0
votes
1answer
55 views

Why use apache spark instead of just a normal worker?

Why should someone use a stream processing engine like Apache Spark, Flink, Hadoop instead of just a normal backend worker which works on something and returns the results as soon as it's done? ...
6
votes
3answers
422 views

Why do you want to avoid flushing stdout?

I stumbled upon a question in Codereview, and in one answer the feedback was to avoid std::endl because it flushes the stream. The full quote is: I'd advise avoiding std::endl in general. Along ...
1
vote
1answer
25 views

how simulate sequential consumers in Kafka

I have kafka consumer that get data from kafka and do some process if some circumstances meet and send it for another consumer to do it's job. How can i build something like this? One way I think ...
1
vote
1answer
77 views

Backend architecture for a datastore with real-time events?

I need to build a data store for transactions that meets the following requirements: A 'transaction' is effectively a state machine which moves through a number of statuses during its lifecycle and ...
0
votes
2answers
124 views

Ideal Interprocess Communication

So, there is a workflow I feel needs to exist. It may already exist, and I don't know about it. Consider the Linux command line workflow, the "motto" of Linux, more or less. You have a bunch of ...
-3
votes
1answer
360 views

Short-circuiting an infinite Java 8 stream

Assume You have some POJO Animal like public class Animal { // some fields } You have some enum AnimalType like public enum AnimalType { // some animal types } You have some method that given an ...
0
votes
0answers
41 views

First DSP project: Amplify digital audio input on Windows 7 on real-time

I want to learn bit about digital signal processing (DSP) with software. As the introduction to the field, I want to make simple real-time PC microphone amplifier, which amplifies the digital ...
0
votes
1answer
43 views

Best practices for reading dynamic byte streams: is line-by-line comparison with a mask the best way?

I receive sensor data as a binary stream of bytes. This stream is not always the same length, and does not include the same data set each time. If the sensor did not send a field, it is simply absent, ...
2
votes
2answers
135 views

Receive data in random order, send in order

I am facing an issue where I have a data stream that sends unordered data. I'm trying to find a way to receive the data in random order, but send it in order. As an example, I'll receive object4 and ...
6
votes
2answers
99 views

Are there any techniques for detecting redundancies a stream of changes to a filesystem?

I'm working on a file-synchronization client that currently produces a stream of changes to the underlying filesystem. That is a stream of create/update/move/delete events is produced for each ...
0
votes
1answer
207 views

Should I produce output with yield or stream.write in Python 3?

I am working on a Python 3 program that will among other things write data to text files in a number of formats. I have one function per format, generating the data to be written. I see two possible ...
-1
votes
1answer
45 views

node server structure and technologies - support continuous and high volume (amount, not size) data transfare

I'm having trouble planning the structure of my server side workflow and the technologies I should use. The basic structure and tasks are: Now, things to consider: 1.the server listens to multiple "...
1
vote
1answer
185 views

Why is the sorted() method on Stream not named sort()?

Why is the sorted() method on Stream not named sort()? list.stream().sort()
0
votes
1answer
764 views

What are reactive streams and what's so special in them?

I browsed Wikipedia and encountered a new feature of Java 1.9: Reactive streams. Unfortunately, the linked Wikipedia page doesn't help much to understand what those reactive streams are and what's so ...
3
votes
2answers
246 views

Are .NET Streams like 'real' streams?

To me a stream implies that I should be able to: Put things in one end and receive it from the other end in the same order. Do these things at the same time. i.e. continuously be adding to the top ...
0
votes
1answer
2k views

Usage of For-each loop vs functional operation vs Java8 streams

What are the gains in actual computing speed (verbosity of code being put aside) and for which case would each be recommended for, as a good design pattern? I would like to know the above in general ...
9
votes
2answers
4k views

Traditional Message Brokers and Streaming Data

According to the Kafka site: "Kakfa is used for building real-time data pipelines and streaming apps." Searching the internet far and wide, I've found the following generally-accepted definition ...
1
vote
4answers
419 views

What is a “streaming app”, exactly?

I'm trying to get past all the hipster, pie-in-the-sky buzzwording and address a very simple, fundamental question: What is a streaming application? According to the Kafka site "Kafka is used for ...
1
vote
1answer
166 views

Moving from Batch to Streaming architecture using a Graph datastructure

The use case I am trying to solve is, assigning millions of users to their groups/segments. I have thousands of different criteria from which buckets of users are created. E.g. bucket criteria: All ...
5
votes
2answers
518 views

Differences between streams and iterators in C++?

I'm working on code which will provide a generic C++ interface to relational datasets. Some of the data will be in tables in memory (e.g. std::set or std::vector), but some of it will be computed - ...
2
votes
5answers
217 views

Algorithm to limit the speed of processing items

I need to process a large number of data items, and I need to restrict the speed at which they are processed. For example, not more than 20 items per minute. I've thought about an algorithm for that, ...
1
vote
0answers
431 views

Streaming architecture (aka Kappa) and different data types

I'm trying to check my understanding of the kappa architecture. The main point I am trying to resolve is how, if at all, one unifies the processing of data under analysis and other pieces of data, ...
1
vote
0answers
499 views

Async stream enrichment with a message queue (e.g. Kafka)

I would like to enrich some streaming data, with one service per enrichment, possibly each service written in a different language and managed/deployed separately. Each service would consume some ...
4
votes
3answers
144 views

Could one swap the direction of >> and << overloads to define them as class members?

Some operators cannot be overloaded as class members. One such example are the bitwise shift operators used for streams see here for example. The reason for that (as far as I understand it) seems to ...
1
vote
2answers
1k views

Video streaming through Ajax calls

These days I have seen couple of sites (including Youtube) that stream videos through Ajax calls (XHR) in few MBs or even less than MB. For an example of what I said see the highlighted lines in the ...
0
votes
1answer
98 views

Where should I store stream wide metadata in stream processing?

I am building a stream processing architecture and was wondering what to do with metadata about the stream. For instance, every message of data coming from a source has the same attribution, as it ...
9
votes
1answer
2k views

How does sorting with java 8 stream work under the hood?

When I call Stream.sort(..) is there a new array of elements created and the stream iterates over the newly created sorted array? In other words, how Java 8 Stream does sort under the hood?
5
votes
1answer
392 views

Why does the C stdio 'ungetc' function exist?

In the C programming language (and many subsequent languages that either directly interfaced with or built a facsimile of the C's Standard IO functions), there exists a function called ungetc: int ...
4
votes
1answer
310 views

Architecture for Real Time data processing

I'm looking at building out the architecture for the following, and wanted to see what others think about it. Assume that the system is running some non-trivial algorithm (so it's not simply a sum ...
0
votes
0answers
46 views

how do I compute the difference given a bidirectional stream of numbers using stream processing?

I want to compute the difference given a bidirectional stream of numbers using stream processing. I'm considering using Apache Spark or a similar technology. To put it more concrete say Bank A is ...
0
votes
1answer
5k views

Base64 decoded image file size

I know base64 encoded binary data is equal to 1.37 times the original data according to Wikipedia. What if you decode that string with php function base64_decode($encoded_string); and then make ...
1
vote
1answer
481 views

Handling responses from an outputstream

I have an InputStream and an OutputStream to a server. I can send a bunch of JSON requests over the wire and get a stream of responses back from this server. From this, I'd like to build a wrapper ...
0
votes
1answer
188 views

Understanding streams

I have started learning streaming APIs and I found one of the good documentations here. There was a comparison given by author to demonstrate the effectiveness of the streams. var http = require('...
0
votes
2answers
285 views

How do you deal with a mutable buffer in a functional space?

I have a stream parsing method in C# that reads portions of a protocol frame from the STOMP protocol; the specification isn't the important part for the question though. What is important is that I ...
1
vote
3answers
351 views

Single exit point meets Java streams

We currently have a sensitive discussion going in the company that touches on a couple of old primary opinion based discussions. Nevertheless, I would like to discuss the case in this forum to ...
1
vote
2answers
52 views

Sending Photo from Mobile to Mobile via Chunking Bytes versus Streaming

I need to send a photo from one mobile phone to another. Currently, I'm splitting the image up into bytes before sending and then reassembling the image on the second mobile. This works fine, but if ...
4
votes
1answer
1k views

Is it better not to .close() inside API method body when passed a Reader or InputStream?

There are plenty of similar questions that have been answered. Example here. However, they all have both the Reader and the InputStream within the same scope or method body, and so suggest to close ...
31
votes
3answers
12k views

Is it an antipattern to use peek() to modify a stream element?

Suppose I have a stream of Things and I want to "enrich" them mid stream, I can use peek() to do this, eg: streamOfThings.peek(this::thingMutator).forEach(this::someConsumer); Assume that mutating ...
3
votes
3answers
195 views

Streams: what has happened after instantiation but before reading?

I'm trying to grok streams - my world is C# but I suspect the principles are general. I understand the general principle of reading/writing bytes from/to a store. However, what I don't understand, ...
2
votes
1answer
153 views

Upload, convert, and store multiple different product data feeds into a database

I'm not sure if this right location for this question, but here it is. I currently have about 30 different CSV or TXT product data feeds. I am trying to build a tool to 1) upload each feed 2) ...
2
votes
1answer
920 views

Feeding std::streambuf for Non-Blocking std::istream Read

I want to write small library to guide a child process for a specific needs (not trying to reinvent bicycle here). My needs includes reading from child stdout and stderr. I want to implement this as ...
5
votes
1answer
1k views

How is one supposed to deal with the intermediate buffer of DataReader class?

Summary I am developing a WAV file format reader under WinRT, and for this I need to read random amounts of structs consisting of fundamental types such as int, uint, float and so on. Back in ...
38
votes
3answers
15k views

Why should I use “functional operations” instead of a for loop?

for (Canvas canvas : list) { } NetBeans suggests me to use "functional operations": list.stream().forEach((canvas) -> { }); But why is this preferred? If anything, it is harder to read and ...
3
votes
1answer
101 views

Is web sockets the best solution for my problem?

I have created a Slackbot that integrates with Submittable.com. For example, I can trigger a response that displays all current submissions in Slack. I now need my Slackbot to post in Slack whenever ...
0
votes
0answers
320 views

Calculating audio delay to eliminate echo in VoIP

In this issue, the microphone at point A records audio, compresses it, and sends it to the server at point B somewhere "in the cloud." The point B server then figures out what rooms the recorded data ...
2
votes
2answers
11k views

Know file size with a base64 string

I do not have very clear what a base64 string is or how it is related to a file. Right now I have a website that uploads an image as a base64 string to my server. This string is converted back into a ...
8
votes
1answer
295 views

What type of buffer should I implement for a one-way streaming audio device?

I'm working on a project where audio data is streamed to a device. The audio data is encoded via opus and streamed at 20 ms payloads at a time. The streaming is done via TCP to avoid packet loss ...
19
votes
3answers
805 views

Is it a sane thing to return Streams wherever we would normally return Collections?

While developing my API that is not tied to any legacy code, I often find myself writing methods that are purely Streams pipeline terminated by collecting the results. Like this one: ImmutableSet<...
0
votes
2answers
253 views

MJRTY - A Fast Majority Vote Algorithm isn't work [closed]

The MJRTY algorithm sets out to solve the problem of finding the majority element in a stream (an element comprising more than 50% of the stream). Moore proposed to solve this by using only 2 pieces ...
0
votes
1answer
382 views

Question on RC4 algorithm

I started reading RC4 from a book and was not able to understand some phrases correctly. The RC4 algorithm is remarkably simple and easy to understand. A variable length key of from 1 to 256 bytes ...