Questions tagged [apache-kafka]
Apache Kafka is publish-subscribe messaging rethought as a distributed commit log.
87
questions
0
votes
0
answers
24
views
Handing skewed processing time of events in a streaming application
I have a streaming application (written over spark/storm/whatever does not matter). Kafka is used as a source of stream events. Now there are some events that take significantly larger resource (time, ...
- 111
1
vote
0
answers
55
views
KRaft design architecture - Separation of Controllers and Brokers
We want to install Kafka Kraft cluster on 35 machines ( machines are DELL physical machines - type R760 And OS is Linux RHEL 8.6 version )
Since we are dealing with Kafka production cluster , we want ...
- 111
0
votes
1
answer
75
views
How to handle aggregations of out-of-order events in an event stream?
I'm interested in integrating with an external system which uses webhooks to notify clients of events. The system is very similar to Stripe - the REST endpoints have rate limits to avoid undue polling ...
0
votes
0
answers
36
views
Using messaging and rest to communicate services at the same time
I have two microservices, USERS and PRODUCTS, the second one creates objects referred to first one so it needs first to check that exists
First I though in a redis cache, but then I realize that for ...
1
vote
2
answers
323
views
How to design inter-service communication with notifications microservice?
I'm creating an application based on the microservice architecture and stuck on notifications microservice design. Basically it should be a service that notifies users about events happening in the ...
- 111
0
votes
2
answers
1k
views
Is Kafka an event bus or a publisher subscriber (pubsub)pattern?
Building on this answer here, and its comments it entails that subscribers need to know and locate the publishers in a traditional pub/sub system. It also entails that publishers need to live at least ...
- 101
0
votes
0
answers
52
views
Kafka : Is sending some payload data (identifier) in headers considered a good practive?
I am wondering about some architectonical decisions in our system.
There are situations where we are sending import business events through kafka which should be performed in every case.
All the cases ...
- 1
0
votes
1
answer
94
views
What pattern should I use for aggregating different types of events in Apache Kafka?
I'm trying to use Kafka for my event handling, but I'm struggling a bit to understand what pattern I should use.
My use case is this:
One consumer waits for one type of event.
When that event is ...
0
votes
0
answers
67
views
"Select All" in Table is now sending too much data to Server via API - Alternative Ways to Send to Server?
Outline of the current architecture of our web app outlining the issue I'm seeing
Client-side app is React, talking to a server running the Play! framework via an API.
On the page is a table that ...
- 101
0
votes
2
answers
133
views
Message broker design pattern best practice [closed]
I'm designing the architecture of a platform to introduce a message broker in an existing data collection web application.
This web application is currently used to upload data from excel files that ...
- 109
1
vote
1
answer
70
views
Kafka on multiple instances of ec2
I am new to Kafka, trying to do a project. Wanted to do it as it would be in real life example, but I am kinda confused. While searching thru the internet I found that if I want to have 3 brokers and ...
0
votes
0
answers
79
views
Can Apache Pinot be used as a general purpose read model?
It's clear that Apache Pinot is intended to be used as an analytics datastore but I can't seem to find anything that explains why it can't be used as a general-purpose read model.
For instance, let's ...
- 101
1
vote
0
answers
76
views
Micro-Services: How to avoid repeated polling of the same event from Kafka
I have an Micro-services where we get the event messages from a Kafka messaging system. The event message contains the orders information, quantity, the mode of package and delivery location. At a ...
- 111
2
votes
1
answer
292
views
Reprocessing out of order events in a stateful stream
Say that you have a monitoring application that reports downtimes for a large number of remote systems (think IOT). A monitoring daemon polls the remote system and reports that status (on/off) via an ...
- 746
0
votes
1
answer
296
views
How to integrate a Python server with Kafka?
I have the request to integrate my Python server application (Flask/waitress) with Kafka.
That means that it should frequently poll a certain Kafka topic (on the order of minutes) and process all new ...
- 101
2
votes
2
answers
750
views
Is Kafka a good idea for connecting a REST API with Worker which runs large computations per job in a queue?
We currently have a REST API service (let's call it as A) written in Python which triggers various heavy calculation jobs for Worker (W) written in Python as well. Both services are interconnected ...
- 73
2
votes
1
answer
138
views
Data Ingest Architecture Advice
I have a requirement where we need to collect N different events and store them for analysis. I am having trouble coming up with a general architecture for this.
FINAL REQUIREMENTS
The end goal of the ...
- 29
0
votes
1
answer
186
views
Can messages from single partition be forwarded to multiple partitions to increase parallel processing in Kafka?
I'm wondering if the following approach can be used in order to increase parallel processing of Kafka messages. Suppose there's a topic with N partitions. At some point the system hits a bottleneck in ...
- 155
1
vote
1
answer
931
views
Handling Errors in a Microservice architecture and asynchronous message queue
When inter-communicating through http requests between Microservices it is quite easy to reply with error messages to users, you can just catch the error of a different Service and forward it to the ...
- 129
0
votes
1
answer
1k
views
C# Kafka Consumer design
I have a C#-based system that works relatively well. It reads data from kafka, processes the data and then writes it out to MSSQL.
The kafka topic has been partitioned into 10 partitions - and after 2 ...
0
votes
2
answers
277
views
Event driven architecture conditional logic
As this article explains the way to deal with persistent state in an event log driven architecture is to have some "microservice" subscribed to events and be responsible for maintaining that ...
-2
votes
1
answer
104
views
Backend microservice design
I'm currently designing a new online platform where the frontend will be Angular using Angular Universal and the backend will be a sort of Node.js clustered microservices.
The question itself is what ...
0
votes
2
answers
200
views
How to publish single event from multiple services
Let's say I have a system that parses network traffic, and generates event of things that happened.
example:
ip1 was talking to ip2
ip2 was talking to ip3
etc
Now let's say that these events are ...
- 101
0
votes
0
answers
104
views
Design : How to divide the responsibility in micro services
I have two micro services built using spring boot,
One micro service is: "payment" Which handles actions related to payments.
Like creating a payment link.
Getting the latest status of ...
- 101
0
votes
1
answer
185
views
Solution architecture for a Kafka streaming to website system
Not sure if this is the correct forum for this question - but I could really use some advice.
I need to design a system (within certain constraints).
The system must implement the following logic:
A ...
0
votes
1
answer
207
views
What limits throughput for a single topic in Kafka, Pulsar, RabbitMQ?
I am trying to assess which of Apache Kafka, RabbitMQ, and Apache Pulsar will meet my throughput requirements for a system with multiple producers and multiple consumers for a single topic.
How can ...
- 119
2
votes
2
answers
1k
views
What is the advantage of Kafka connector to MongoDB over using MongoDB change streams?
I need to stream changes from my MongoDB instance to Elasticsearch. I also need to perform complex transformations on changed documents. After doing some research I narrowed my options down to 2:
...
- 155
1
vote
1
answer
1k
views
Kafka consumers - can/should it be a REST API also?
We are starting a journey in our company to use Kafka as our event backbone for both streaming data and for an event driven architecture. I foresee a pattern where we would want to consume events off ...
- 111
0
votes
1
answer
291
views
Mixing an API server and a background worker
What are your thoughts on having a service/process be both an API server and a background worker, consuming messages off a queue like SQS?
I have typically kept these two separate to keep single ...
- 111
-1
votes
2
answers
122
views
Do I need to use Apache Kafka alongside another DBMS in my use case?
I have a project that relies on getting data from IoT devices.
then I display the data analytics using a graph (in real-time) and also the state of the machines (are they working or not and the reason ...
7
votes
2
answers
12k
views
Is Kafka needed in a realtime chat application?
I'm developing a realtime chat application with an Angular frontend and Java backend. I've found a couple of examples that resemble what I am trying to achieve, such as:
https://medium.com/swlh/...
- 211
0
votes
1
answer
81
views
What to process in a Kafka broker vs in a Kafka Streams client application?
Using Kafka I understand that it makes little sense to simply pass events in and out of a Kafka cluster, and that the real benefit comes when doing some processing in the events received in the ...
- 101
1
vote
1
answer
300
views
How would a Search API microservice fit in an event driven architecture?
I apologize if this sounds really strange. I am very new to System Design and have just been reading up on Event driven systems. To gain better understanding and to perfect my thought process I ...
- 157
0
votes
0
answers
102
views
Is queue is the right choice for server request retries
I have server A that received millions of requests per day and for every request it notifies server B about it.
sometimes server B is down (maintenance, bugs etc..) so requests from server A got lost.
...
- 109
1
vote
1
answer
271
views
Options for building initial event sourcing projection when you cannot go back to the beginning of the history
I’m building a microservice which depends on a set of downstream services to collect the data necessary to operate. It is a data delivery system, I’m not source of truth for any data, my system rather ...
0
votes
0
answers
53
views
How to deal with time-based conditions in an event-driven architecture?
The company I work for has a ticketing platform. The backend system consists of a few applications talking to each other via HTTP and through events in Kafka. A ticket has a sell_start_date field ...
- 249
0
votes
0
answers
44
views
Designing sane messaging paradigm for concurrent updates
We have a weird use-case that I need to support in my application and needed some opinions on how to design it.
At a basic level the application allows users to work on a "list" of "tweets" she cares ...
- 109
2
votes
2
answers
330
views
Processing messages in parallel with Kafka & Akka
We have data in Kafka which we need to export to multiple destinations. Each message key is to be exported to one destination.
A destination can be a REST endpoint, a file, a database etc.
Each ...
- 156
1
vote
1
answer
256
views
Designing Kafka topics for secured event driven job scheduling system
Consider the following
Group A
Job A {
Depends on Job B of Group A
Run User -> User1
}
Job B {
Depends on Job C and Job D of Group A
Run User -> User2
}
Job C {
Depends ...
- 119
0
votes
1
answer
161
views
Integrating with real-time data from multiple devices with accelerometer
I am designing a MVP for a simple gamification system for a trampoline park(s).
External company is providing bracelets for customers, then collecting this data in their own web app, where they ...
- 67
2
votes
2
answers
5k
views
Microservice architecture pattern for Batch based system
I have been exploring the microservice architecture for the batch-based system.
Here is our current setup:
Code:
We have 5 systems that are internally connected and they pass data from one system ...
- 79
4
votes
3
answers
864
views
Alternating between Java streams and parallel streams at runtime
This is a question I constantly ask myself when designing a data intensive application: When is it appropriate to use stream() over parallelStream()? Would it make sense to use both? How do I quantify ...
- 263
0
votes
0
answers
59
views
Deciding on Idempotency Approach on Distirbuted Systems (At Kafka Consumer)
The service that is built works on requests reply method using Kafka consumers. This service communicates with external systems .
Whenever an external system requests a message(via KAfka) They had ...
- 167
0
votes
1
answer
796
views
Will kafka consumers differentiate events on the same topic, or get all events on that topic?
How will a kafka consumer differentiate/behave on the type of events that is stored in a given topic ?
Let's say I have a consumer (Consumer1) which is set up to consume from a topic with event of ...
- 743
5
votes
2
answers
2k
views
kafka consumer properties session.timeout.ms vs heartbeat.interval.ms
I am reading Kafka, the definitive guide and I came across the below point for consumers.
heartbeat.interval.ms must be lower than session.timeout.ms, and is usually set to one-third of the timeout ...
- 59
1
vote
0
answers
128
views
Kafka Consumer in Azure
I'm working on a Kafka consumer in C# (using the Confluent C# lib).. and practically all of the documentation/examples online leverage console apps to do the consuming.
So, this is fine for logic ...
- 237
2
votes
1
answer
271
views
How to unit test a service which output is an event with data?
Technologies : NodeJs, Kafka, Mocha
The issue : I have a nodeJS service which given a String returns events with a many data.
How can I test this using Mocha ?
In a simple scenario the service ...
- 1,001
1
vote
0
answers
94
views
Efficient Options to do Stream Joins(with Large Time gaps)
Use Case: I need to join two stream sources(say orders(order_id, order_val) and shipments(shipment_id, shipment_val)) based on an id(order_id = shipment_id) and generate a new event shipment_order(id, ...
- 129
-1
votes
2
answers
152
views
Using Apache Kafka for rare events [closed]
There are multiple microservices which are responsible for storing data dependent from user, e.g. user preferences on some products, user orders, etc. There is a scenario when user is deleted from ...
- 111
1
vote
1
answer
370
views
Is Kafka the right choice to decouple my APIs from the service implementation?
I wan't to decouple my exposed API service from their implementation. So I was planning to use KafKa as a mediation layer between my exposed API and the service implementation by going fully ...
- 231