Questions tagged [apache-kafka]

Apache Kafka is publish-subscribe messaging rethought as a distributed commit log.

Filter by
Sorted by
Tagged with
0
votes
0answers
57 views

Can messages from single partition be forwarded to multiple partitions to increase parallel processing in Kafka?

I'm wondering if the following approach can be used in order to increase parallel processing of Kafka messages. Suppose there's a topic with N partitions. At some point the system hits a bottleneck in ...
0
votes
0answers
36 views

design for the order processing system

We have REST endpoint that that works like this Retrieve payload from request Generated unique id for payload and order_status and enrichment_status to PENDING Store payload in Redis Perform request ...
1
vote
1answer
116 views

Handling Errors in a Microservice architecture and asynchronous message queue

When inter-communicating through http requests between Microservices it is quite easy to reply with error messages to users, you can just catch the error of a different Service and forward it to the ...
0
votes
1answer
84 views

C# Kafka Consumer design

I have a C#-based system that works relatively well. It reads data from kafka, processes the data and then writes it out to MSSQL. The kafka topic has been partitioned into 10 partitions - and after 2 ...
0
votes
2answers
96 views

Event driven architecture conditional logic

As this article explains the way to deal with persistent state in an event log driven architecture is to have some "microservice" subscribed to events and be responsible for maintaining that ...
-2
votes
1answer
64 views

Backend microservice design

I'm currently designing a new online platform where the frontend will be Angular using Angular Universal and the backend will be a sort of Node.js clustered microservices. The question itself is what ...
0
votes
1answer
71 views

How to publish single event from multiple services

Let's say I have a system that parses network traffic, and generates event of things that happened. example: ip1 was talking to ip2 ip2 was talking to ip3 etc Now let's say that these events are ...
0
votes
0answers
39 views

Design : How to divide the responsibility in micro services

I have two micro services built using spring boot, One micro service is: "payment" Which handles actions related to payments. Like creating a payment link. Getting the latest status of ...
0
votes
1answer
55 views

Solution architecture for a Kafka streaming to website system

Not sure if this is the correct forum for this question - but I could really use some advice. I need to design a system (within certain constraints). The system must implement the following logic: A ...
0
votes
1answer
72 views

What limits throughput for a single topic in Kafka, Pulsar, RabbitMQ?

I am trying to assess which of Apache Kafka, RabbitMQ, and Apache Pulsar will meet my throughput requirements for a system with multiple producers and multiple consumers for a single topic. How can ...
2
votes
2answers
340 views

What is the advantage of Kafka connector to MongoDB over using MongoDB change streams?

I need to stream changes from my MongoDB instance to Elasticsearch. I also need to perform complex transformations on changed documents. After doing some research I narrowed my options down to 2: ...
1
vote
1answer
149 views

Kafka consumers - can/should it be a REST API also?

We are starting a journey in our company to use Kafka as our event backbone for both streaming data and for an event driven architecture. I foresee a pattern where we would want to consume events off ...
0
votes
1answer
44 views

Mixing an API server and a background worker

What are your thoughts on having a service/process be both an API server and a background worker, consuming messages off a queue like SQS? I have typically kept these two separate to keep single ...
-1
votes
2answers
76 views

Do I need to use Apache Kafka alongside another DBMS in my use case?

I have a project that relies on getting data from IoT devices. then I display the data analytics using a graph (in real-time) and also the state of the machines (are they working or not and the reason ...
7
votes
2answers
5k views

Is Kafka needed in a realtime chat application?

I'm developing a realtime chat application with an Angular frontend and Java backend. I've found a couple of examples that resemble what I am trying to achieve, such as: https://medium.com/swlh/...
0
votes
1answer
36 views

What to process in a Kafka broker vs in a Kafka Streams client application?

Using Kafka I understand that it makes little sense to simply pass events in and out of a Kafka cluster, and that the real benefit comes when doing some processing in the events received in the ...
1
vote
1answer
196 views

How would a Search API microservice fit in an event driven architecture?

I apologize if this sounds really strange. I am very new to System Design and have just been reading up on Event driven systems. To gain better understanding and to perfect my thought process I ...
0
votes
0answers
87 views

Is queue is the right choice for server request retries

I have server A that received millions of requests per day and for every request it notifies server B about it. sometimes server B is down (maintenance, bugs etc..) so requests from server A got lost. ...
1
vote
1answer
140 views

Options for building initial event sourcing projection when you cannot go back to the beginning of the history

I’m building a microservice which depends on a set of downstream services to collect the data necessary to operate. It is a data delivery system, I’m not source of truth for any data, my system rather ...
0
votes
0answers
40 views

How to deal with time-based conditions in an event-driven architecture?

The company I work for has a ticketing platform. The backend system consists of a few applications talking to each other via HTTP and through events in Kafka. A ticket has a sell_start_date field ...
0
votes
0answers
43 views

Designing sane messaging paradigm for concurrent updates

We have a weird use-case that I need to support in my application and needed some opinions on how to design it. At a basic level the application allows users to work on a "list" of "tweets" she cares ...
2
votes
2answers
158 views

Processing messages in parallel with Kafka & Akka

We have data in Kafka which we need to export to multiple destinations. Each message key is to be exported to one destination. A destination can be a REST endpoint, a file, a database etc. Each ...
1
vote
1answer
167 views

Designing Kafka topics for secured event driven job scheduling system

Consider the following Group A Job A { Depends on Job B of Group A Run User -> User1 } Job B { Depends on Job C and Job D of Group A Run User -> User2 } Job C { Depends ...
0
votes
1answer
150 views

Integrating with real-time data from multiple devices with accelerometer

I am designing a MVP for a simple gamification system for a trampoline park(s). External company is providing bracelets for customers, then collecting this data in their own web app, where they ...
2
votes
2answers
3k views

Microservice architecture pattern for Batch based system

I have been exploring the microservice architecture for the batch-based system. Here is our current setup: Code: We have 5 systems that are internally connected and they pass data from one system ...
2
votes
3answers
451 views

Alternating between Java streams and parallel streams at runtime

This is a question I constantly ask myself when designing a data intensive application: When is it appropriate to use stream() over parallelStream()? Would it make sense to use both? How do I quantify ...
0
votes
0answers
43 views

Deciding on Idempotency Approach on Distirbuted Systems (At Kafka Consumer)

The service that is built works on requests reply method using Kafka consumers. This service communicates with external systems . Whenever an external system requests a message(via KAfka) They had ...
0
votes
1answer
353 views

Will kafka consumers differentiate events on the same topic, or get all events on that topic?

How will a kafka consumer differentiate/behave on the type of events that is stored in a given topic ? Let's say I have a consumer (Consumer1) which is set up to consume from a topic with event of ...
4
votes
2answers
764 views

kafka consumer properties session.timeout.ms vs heartbeat.interval.ms

I am reading Kafka, the definitive guide and I came across the below point for consumers. heartbeat.interval.ms must be lower than session.timeout.ms, and is usually set to one-third of the timeout ...
1
vote
0answers
55 views

Kafka Consumer in Azure

I'm working on a Kafka consumer in C# (using the Confluent C# lib).. and practically all of the documentation/examples online leverage console apps to do the consuming. So, this is fine for logic ...
2
votes
1answer
172 views

How to unit test a service which output is an event with data?

Technologies : NodeJs, Kafka, Mocha The issue : I have a nodeJS service which given a String returns events with a many data. How can I test this using Mocha ? In a simple scenario the service ...
1
vote
0answers
54 views

Efficient Options to do Stream Joins(with Large Time gaps)

Use Case: I need to join two stream sources(say orders(order_id, order_val) and shipments(shipment_id, shipment_val)) based on an id(order_id = shipment_id) and generate a new event shipment_order(id, ...
-1
votes
2answers
101 views

Using Apache Kafka for rare events [closed]

There are multiple microservices which are responsible for storing data dependent from user, e.g. user preferences on some products, user orders, etc. There is a scenario when user is deleted from ...
1
vote
1answer
170 views

Is Kafka the right choice to decouple my APIs from the service implementation?

I wan't to decouple my exposed API service from their implementation. So I was planning to use KafKa as a mediation layer between my exposed API and the service implementation by going fully ...
0
votes
1answer
152 views

Kafka cluster workflow working?

I have read lot of articles on google how kafka cluster works. Trying to summarizing my understanding here . Please let me know if any point is not correct here? Producer Side Say Kafka cluster ...
0
votes
2answers
3k views

Maintaining order of events with multiple consumers in a typical pub sub scenario

I am using Kafka. I am developing a simple e-commerce solution. I have a non-scalable catalog admin portal where products, categories, attributes, variants of products, channels, etc are updated. For ...
2
votes
1answer
2k views

Service Bus or Event Stream for Event-Driven Architecture

I'm creating a set of Microservices following Event-Driven Architecture using Event-Carried State Transfer to move state from one service to another. Given the following services; Order service ...
3
votes
1answer
185 views

Data processing pipeline design for processing data

I have a use case for which I need to build a data processing pipeline Customer contact leads data coming from different data sources like csv, data base, api has to be first mapped to a universal ...
0
votes
1answer
102 views

how simulate sequential consumers in Kafka

I have kafka consumer that get data from kafka and do some process if some circumstances meet and send it for another consumer to do it's job. How can i build something like this? One way I think ...
9
votes
1answer
2k views

Data durability guarantees in Kafka

Is it wise to use kafka as the 'source of truth' for mission critical data? The setup is: kafka is the underlying source-of-truth for the data. -querying is done on caches (I.e. Redis, ktables) ...
2
votes
3answers
443 views

How to replace parallel async calls between micro services with event based communication

A service A needs to access services B, C and D to process a request. Currently this is implemented as parallel async calls from A to B, C and D. Once all of them has responded, service A follows some ...
0
votes
1answer
2k views

Processing only once the same message produced by two producers

If I have two different producers that could produce the same message for a Kafka broker, how can I ensure that only one of the two message occurrences gets processed? Is the only way to have an ...
0
votes
1answer
31 views

Retry mechanism - 3rd party provider

I have the scenario below and what I want to do is to build a retry mechanism to retrieve a status. So, what we have is a payment system and we want to call a 3rd party provided periodically in order ...
2
votes
1answer
506 views

Microservices loose coupling using distributed business logic or centralized?

How do we achieve loose coupling in the below scenario: N microservices (let's call them Callers) needs a logic or business( let's call it Worker) to be executed with different contracts. Although ...
2
votes
1answer
824 views

Message Broker vs. iPaaS

Does anyone have suggestions/recommendations/guidance they can offer on when to use a message broker like RabbitMQ/Kafka vs. an iPaaS solution like Mulesoft/Jitterbit/Dell Boomi/Workato? Can they ...
0
votes
1answer
323 views

Co-relating messages in Kafka

Context We are building an application that acts like a hub for messages coming in from multiple Producers and it assigns to Consumers based on certain criteria. Need is to have the design highly ...
0
votes
1answer
49 views

Evaluate http reqres vs websockets for geolocation based broadcasts

My use case is that I have a real-time broadcast app where users get feed items based on a geolocation, i.e. a user will receive posts that were broadcasted from within a user-defined radius where (s)...
1
vote
1answer
81 views

What does 'data coverage' mean when talking about ETL processes?

I was watching this talk about ETL's shortcomings and the solutions provided by the Kafka platform but I don't quite understand what the speaker is referring to when she says ETL tools have been ...
1
vote
1answer
244 views

Building a real-time API on a Kafka/Kinesis-centric structure

I am a beginner to the event-driven data architecture using Kafka / Kinesis as the centrepiece (currently Kinesis) and I have some questions regarding how to build a WebSocket API on such a structure. ...
7
votes
1answer
470 views

Is it appropriate to say that RabbitMQ and Apache Kafka solve similar problems?

I have used RabbitMQ but I haven't used Apache Kafka. Is it a similar problem that these products solve, or is there no connection?