Questions tagged [stream-processing]

Stream processing is consuming an input stream or sequence of bytes with some format, processing the data as it arrives, and translating it into another format in a related output stream.

Filter by
Sorted by
Tagged with
1 vote
0 answers
703 views

Streaming architecture (aka Kappa) and different data types

I'm trying to check my understanding of the kappa architecture. The main point I am trying to resolve is how, if at all, one unifies the processing of data under analysis and other pieces of data, ...
user avatar
  • 111
1 vote
0 answers
619 views

Async stream enrichment with a message queue (e.g. Kafka)

I would like to enrich some streaming data, with one service per enrichment, possibly each service written in a different language and managed/deployed separately. Each service would consume some ...
user avatar
  • 19
4 votes
3 answers
181 views

Could one swap the direction of >> and << overloads to define them as class members?

Some operators cannot be overloaded as class members. One such example are the bitwise shift operators used for streams see here for example. The reason for that (as far as I understand it) seems to ...
user avatar
  • 3,496
1 vote
2 answers
3k views

Video streaming through Ajax calls

These days I have seen couple of sites (including Youtube) that stream videos through Ajax calls (XHR) in few MBs or even less than MB. For an example of what I said see the highlighted lines in the ...
user avatar
0 votes
1 answer
123 views

Where should I store stream wide metadata in stream processing?

I am building a stream processing architecture and was wondering what to do with metadata about the stream. For instance, every message of data coming from a source has the same attribution, as it ...
user avatar
13 votes
1 answer
4k views

How does sorting with java 8 stream work under the hood?

When I call Stream.sort(..) is there a new array of elements created and the stream iterates over the newly created sorted array? In other words, how Java 8 Stream does sort under the hood?
user avatar
  • 2,939
6 votes
1 answer
698 views

Why does the C stdio 'ungetc' function exist?

In the C programming language (and many subsequent languages that either directly interfaced with or built a facsimile of the C's Standard IO functions), there exists a function called ungetc: int ...
user avatar
  • 4,449
4 votes
1 answer
386 views

Architecture for Real Time data processing

I'm looking at building out the architecture for the following, and wanted to see what others think about it. Assume that the system is running some non-trivial algorithm (so it's not simply a sum ...
user avatar
  • 141
0 votes
1 answer
9k views

Base64 decoded image file size

I know base64 encoded binary data is equal to 1.37 times the original data according to Wikipedia. What if you decode that string with php function base64_decode($encoded_string); and then make ...
user avatar
  • 103
1 vote
1 answer
833 views

Handling responses from an outputstream

I have an InputStream and an OutputStream to a server. I can send a bunch of JSON requests over the wire and get a stream of responses back from this server. From this, I'd like to build a wrapper ...
user avatar
0 votes
1 answer
252 views

Understanding streams

I have started learning streaming APIs and I found one of the good documentations here. There was a comparison given by author to demonstrate the effectiveness of the streams. var http = require('...
user avatar
  • 2,106
0 votes
2 answers
412 views

How do you deal with a mutable buffer in a functional space?

I have a stream parsing method in C# that reads portions of a protocol frame from the STOMP protocol; the specification isn't the important part for the question though. What is important is that I ...
user avatar
  • 208
1 vote
3 answers
565 views

Single exit point meets Java streams

We currently have a sensitive discussion going in the company that touches on a couple of old primary opinion based discussions. Nevertheless, I would like to discuss the case in this forum to ...
user avatar
1 vote
2 answers
62 views

Sending Photo from Mobile to Mobile via Chunking Bytes versus Streaming

I need to send a photo from one mobile phone to another. Currently, I'm splitting the image up into bytes before sending and then reassembling the image on the second mobile. This works fine, but if ...
user avatar
  • 189
4 votes
1 answer
2k views

Is it better not to .close() inside API method body when passed a Reader or InputStream?

There are plenty of similar questions that have been answered. Example here. However, they all have both the Reader and the InputStream within the same scope or method body, and so suggest to close ...
user avatar
  • 576
54 votes
3 answers
33k views

Is it an antipattern to use peek() to modify a stream element?

Suppose I have a stream of Things and I want to "enrich" them mid stream, I can use peek() to do this, eg: streamOfThings.peek(this::thingMutator).forEach(this::someConsumer); Assume that ...
user avatar
  • 1,836
3 votes
3 answers
407 views

Streams: what has happened after instantiation but before reading?

I'm trying to grok streams - my world is C# but I suspect the principles are general. I understand the general principle of reading/writing bytes from/to a store. However, what I don't understand, ...
user avatar
2 votes
1 answer
171 views

Upload, convert, and store multiple different product data feeds into a database

I'm not sure if this right location for this question, but here it is. I currently have about 30 different CSV or TXT product data feeds. I am trying to build a tool to 1) upload each feed 2) ...
user avatar
2 votes
1 answer
1k views

Feeding std::streambuf for Non-Blocking std::istream Read

I want to write small library to guide a child process for a specific needs (not trying to reinvent bicycle here). My needs includes reading from child stdout and stderr. I want to implement this as ...
user avatar
6 votes
1 answer
2k views

How is one supposed to deal with the intermediate buffer of DataReader class?

Summary I am developing a WAV file format reader under WinRT, and for this I need to read random amounts of structs consisting of fundamental types such as int, uint, float and so on. Back in ...
user avatar
  • 727
50 votes
4 answers
21k views

Why should I use "functional operations" instead of a for loop?

for (Canvas canvas : list) { } NetBeans suggests me to use "functional operations": list.stream().forEach((canvas) -> { }); But why is this preferred? If anything, it is harder to read and ...
user avatar
  • 3,831
3 votes
1 answer
112 views

Is web sockets the best solution for my problem?

I have created a Slackbot that integrates with Submittable.com. For example, I can trigger a response that displays all current submissions in Slack. I now need my Slackbot to post in Slack whenever ...
user avatar
  • 133
0 votes
1 answer
381 views

Calculating audio delay to eliminate echo in VoIP

In this issue, the microphone at point A records audio, compresses it, and sends it to the server at point B somewhere "in the cloud." The point B server then figures out what rooms the recorded data ...
user avatar
11 votes
2 answers
58k views

Know file size with a base64 string

I do not have very clear what a base64 string is or how it is related to a file. Right now I have a website that uploads an image as a base64 string to my server. This string is converted back into a ...
user avatar
8 votes
1 answer
714 views

What type of buffer should I implement for a one-way streaming audio device?

I'm working on a project where audio data is streamed to a device. The audio data is encoded via opus and streamed at 20 ms payloads at a time. The streaming is done via TCP to avoid packet loss ...
user avatar
20 votes
3 answers
901 views

Is it a sane thing to return Streams wherever we would normally return Collections?

While developing my API that is not tied to any legacy code, I often find myself writing methods that are purely Streams pipeline terminated by collecting the results. Like this one: ImmutableSet<...
user avatar
  • 303
0 votes
2 answers
341 views

MJRTY - A Fast Majority Vote Algorithm isn't work [closed]

The MJRTY algorithm sets out to solve the problem of finding the majority element in a stream (an element comprising more than 50% of the stream). Moore proposed to solve this by using only 2 pieces ...
user avatar
  • 103
1 vote
1 answer
515 views

Question on RC4 algorithm

I started reading RC4 from a book and was not able to understand some phrases correctly. The RC4 algorithm is remarkably simple and easy to understand. A variable length key of from 1 to 256 bytes ...
user avatar
  • 135
1 vote
1 answer
438 views

Using streams to connect programming languages

I'm trying to run a computation-heavy program that creates an image on a website. Is it possible to compute in C++ and have an output stream that connects to an input stream in Node.js to display an ...
user avatar
2 votes
1 answer
1k views

Finding a solution for interleaving multiple data messages into one network stream (in QT framework)

I need to find a solution for high throughput network data interleaving in my QT framework based windows application. My application is based on server/client architecture. The client connects to a ...
user avatar
  • 121
41 votes
4 answers
45k views

What is a byte stream actually?

Can anyone explain me what byte stream actually contains? Does it contain bytes (hex data) or binary data or english letters only? I am also confused about the term "raw data". If someone asked me to "...
user avatar

1
2