Questions tagged [apache-kafka]

Apache Kafka is publish-subscribe messaging rethought as a distributed commit log.

Filter by
Sorted by
Tagged with
1
vote
2answers
152 views
+50

Microservice architecture pattern for Batch based system

I have been exploring the microservice architecture for the batch-based system. Here is our current setup: Code: We have 5 systems that are internally connected and they pass data from one system ...
2
votes
2answers
91 views

Alternating between Java streams and parallel streams at runtime

This is a question I constantly ask myself when designing a data intensive application: When is it appropriate to use stream() over parallelStream()? Would it make sense to use both? How do I quantify ...
0
votes
0answers
30 views

Deciding on Idempotency Approach on Distirbuted Systems (At Kafka Consumer)

The service that is built works on requests reply method using Kafka consumers. This service communicates with external systems . Whenever an external system requests a message(via KAfka) They had ...
0
votes
1answer
33 views

Will kafka consumers differentiate events on the same topic, or get all events on that topic?

How will a kafka consumer differentiate/behave on the type of events that is stored in a given topic ? Let's say I have a consumer (Consumer1) which is set up to consume from a topic with event of ...
1
vote
0answers
86 views

kafka consumer properties session.timeout.ms vs heartbeat.interval.ms

I am reading Kafka, the definitive guide and I came across the below point for consumers. heartbeat.interval.ms must be lower than session.timeout.ms, and is usually set to one-third of the timeout ...
1
vote
0answers
22 views

Kafka Consumer in Azure

I'm working on a Kafka consumer in C# (using the Confluent C# lib).. and practically all of the documentation/examples online leverage console apps to do the consuming. So, this is fine for logic ...
3
votes
1answer
94 views

How to unit test a service which output is an event with data?

Technologies : NodeJs, Kafka, Mocha The issue : I have a nodeJS service which given a String returns events with a many data. How can I test this using Mocha ? In a simple scenario the service ...
0
votes
0answers
30 views

Efficient Options to do Stream Joins(with Large Time gaps)

Use Case: I need to join two stream sources(say orders(order_id, order_val) and shipments(shipment_id, shipment_val)) based on an id(order_id = shipment_id) and generate a new event shipment_order(id, ...
-1
votes
2answers
68 views

Using Apache Kafka for rare events [closed]

There are multiple microservices which are responsible for storing data dependent from user, e.g. user preferences on some products, user orders, etc. There is a scenario when user is deleted from ...
0
votes
0answers
24 views

Are message identifiers and offsets the same thing in Kafka?

I am reading up on Kafka and encountered the following on one of the tutorials. The producer sends the following to the broker - unique message identifier, timestamp and a binary payload. The ...
1
vote
1answer
110 views

Is Kafka the right choice to decouple my APIs from the service implementation?

I wan't to decouple my exposed API service from their implementation. So I was planning to use KafKa as a mediation layer between my exposed API and the service implementation by going fully ...
0
votes
0answers
41 views

Synchronize Data From Multiple Data Sources

Our team is trying to build a predictive maintenance system whose task is to look at a set of events and predict whether these events depict a set of known anomalies or not. We are at the design ...
0
votes
0answers
159 views

How to manage microservice workflows with Apache Kafka

Currently, we have REST-based services and we would like to make an architectural shift to decoupled microservices with Kafka. However, there are a couple of issues not clear to me. 1- How can we ...
0
votes
1answer
86 views

Kafka cluster workflow working?

I have read lot of articles on google how kafka cluster works. Trying to summarizing my understanding here . Please let me know if any point is not correct here? Producer Side Say Kafka cluster ...
0
votes
2answers
1k views

Maintaining order of events with multiple consumers in a typical pub sub scenario

I am using Kafka. I am developing a simple e-commerce solution. I have a non-scalable catalog admin portal where products, categories, attributes, variants of products, channels, etc are updated. For ...
1
vote
1answer
594 views

Service Bus or Event Stream for Event-Driven Architecture

I'm creating a set of Microservices following Event-Driven Architecture using Event-Carried State Transfer to move state from one service to another. Given the following services; Order service ...
3
votes
0answers
125 views

Data processing pipeline design for processing data

I have a use case for which I need to build a data processing pipeline Customer contact leads data coming from different data sources like csv, data base, api has to be first mapped to a universal ...
1
vote
1answer
58 views

how simulate sequential consumers in Kafka

I have kafka consumer that get data from kafka and do some process if some circumstances meet and send it for another consumer to do it's job. How can i build something like this? One way I think ...
9
votes
1answer
650 views

Data durability guarantees in Kafka

Is it wise to use kafka as the 'source of truth' for mission critical data? The setup is: kafka is the underlying source-of-truth for the data. -querying is done on caches (I.e. Redis, ktables) ...
2
votes
3answers
281 views

How to replace parallel async calls between micro services with event based communication

A service A needs to access services B, C and D to process a request. Currently this is implemented as parallel async calls from A to B, C and D. Once all of them has responded, service A follows some ...
0
votes
1answer
1k views

Processing only once the same message produced by two producers

If I have two different producers that could produce the same message for a Kafka broker, how can I ensure that only one of the two message occurrences gets processed? Is the only way to have an ...
0
votes
1answer
21 views

Retry mechanism - 3rd party provider

I have the scenario below and what I want to do is to build a retry mechanism to retrieve a status. So, what we have is a payment system and we want to call a 3rd party provided periodically in order ...
2
votes
1answer
209 views

Microservices loose coupling using distributed business logic or centralized?

How do we achieve loose coupling in the below scenario: N microservices (let's call them Callers) needs a logic or business( let's call it Worker) to be executed with different contracts. Although ...
1
vote
1answer
443 views

Message Broker vs. iPaaS

Does anyone have suggestions/recommendations/guidance they can offer on when to use a message broker like RabbitMQ/Kafka vs. an iPaaS solution like Mulesoft/Jitterbit/Dell Boomi/Workato? Can they ...
0
votes
1answer
208 views

Co-relating messages in Kafka

Context We are building an application that acts like a hub for messages coming in from multiple Producers and it assigns to Consumers based on certain criteria. Need is to have the design highly ...
0
votes
1answer
42 views

Evaluate http reqres vs websockets for geolocation based broadcasts

My use case is that I have a real-time broadcast app where users get feed items based on a geolocation, i.e. a user will receive posts that were broadcasted from within a user-defined radius where (s)...
1
vote
1answer
51 views

What does 'data coverage' mean when talking about ETL processes?

I was watching this talk about ETL's shortcomings and the solutions provided by the Kafka platform but I don't quite understand what the speaker is referring to when she says ETL tools have been ...
1
vote
1answer
170 views

Building a real-time API on a Kafka/Kinesis-centric structure

I am a beginner to the event-driven data architecture using Kafka / Kinesis as the centrepiece (currently Kinesis) and I have some questions regarding how to build a WebSocket API on such a structure. ...
6
votes
1answer
353 views

Is it appropriate to say that RabbitMQ and Apache Kafka solve similar problems?

I have used RabbitMQ but I haven't used Apache Kafka. Is it a similar problem that these products solve, or is there no connection?
1
vote
2answers
2k views

How to build a scalable, containerizable and manageable API with GraphQL, Kafka, Docker, Kubernetes and MongoDB?

I have no doubt regarding what each of the technology does. I have tried all of them as well. What I have doubt with is: In case we use GraphQL, wont it act like a single point of failure if the ...
-1
votes
1answer
495 views

Real time Querying on Real-time Data

I have data flowing from Kafka into MongoDB real-time. This is just raw data and various APIs are served using this data in real time. The APIs respond using aggregation queries. However when data to ...
4
votes
0answers
376 views

Can Event Sourcing and an API Management platform leverage each other?

I'm evaluating the use of Kafka to implement Event Sourcing in a microservices environment that already makes use of an API Management platform. I was wondering if it is common and/or considered a ...
3
votes
1answer
172 views

How to organize exchanges of non critical messages between servers in multiple data centers by using Kafka (or other solution)?

We need to organize a way to exchange messages between servers in multiple data centers. Messages are not critical. We just need to be able to send messages from any servers in any data centers to any ...
1
vote
1answer
2k views

How can I handle multi-producer to particular single-consumer in Kafka?

I'm evaluating the Kafka. It provides some nice features which I want to use, especially durability be default and the access to past messages for new consumers, whenever the consumer wants to reach ...
3
votes
0answers
290 views

Designing clickstream analysis?

I have a application where user purchases/click the certain products. I need to design the click stream analysis here which product got clicked how many number of time, user/geographical detail click ...
13
votes
2answers
6k views

Traditional Message Brokers and Streaming Data

According to the Kafka site: "Kakfa is used for building real-time data pipelines and streaming apps." Searching the internet far and wide, I've found the following generally-accepted definition ...
1
vote
4answers
689 views

What is a “streaming app”, exactly?

I'm trying to get past all the hipster, pie-in-the-sky buzzwording and address a very simple, fundamental question: What is a streaming application? According to the Kafka site "Kafka is used for ...
2
votes
1answer
88 views

Task Assigning at a Distributed System

I want to crawl websites and write output to Solr with my application. My crawler uses Kafka to put crawled website object on a topic and Flume will redirect output to Solr. There are multiple ...
2
votes
1answer
1k views

Event Driven Microservice Race Condition

Say we have 4 services, A through D, which communicate (for the most part) through some sort of asynchronous event-driven system. When a new entity is created in A, B & C receive that event. B ...
1
vote
1answer
306 views

Exploiting Apache Spark Data

and sorry if the question seems a bit naive. I'm currently reading tutorials about Kafka & Spark and there's something I can't figure out : how to exploit / expose the data Spark received. Here'...
4
votes
3answers
4k views

Decoupling microservices with gRPC

I'm setting up a microservices architecture, and am confused about how gRPC can loosely-couple services (compared to a pub-sub message service like Kafka). Doesn't the request go directly to the ...
1
vote
4answers
3k views

Web application Architecture for real time app

Usecase: End user places an order on the company website, the order needs to be pushed to company admin's screen with small latency.The company has multiple branches(identified by compId and branchId ...
1
vote
0answers
568 views

Async stream enrichment with a message queue (e.g. Kafka)

I would like to enrich some streaming data, with one service per enrichment, possibly each service written in a different language and managed/deployed separately. Each service would consume some ...
3
votes
2answers
755 views

Two processes in a single docker container or two services connecting to the same db?

I recently started moving a monolithic application to microservices architecture using docker containers. The general idea of the app is: scraping data -> format the data -> save the data to MySQL -...
1
vote
1answer
173 views

#Apache-flink: Stream processing or Batch processing using Flink

I am tasked with redesigning an existing catalog processor and the requirement goes as below Requirement I have 5 to 10 vendors(each vendor can have multiple stores) who would provide me with 'XML' ...
8
votes
2answers
9k views

Event sourcing, replaying and versioning

I am designing a system that uses Event Sourcing, CQRS and microservices. I am lead to understand this isn't an uncommon pattern. A key feature of the service needs to be the ability to rehydrate/...
2
votes
2answers
3k views

Distributed transactions with Kafka

I need to implement a transaction that spans over loosely coupled (SOA+MOM) components. When a particular event is received, FOO and BAR need to do a transactional operation: that is, both the ...
0
votes
2answers
749 views

Send heavy computation jobs from web server to another process and wait for response

I know it is a common practice to send cpu-bounded heavy tasks to background processes, in order to free a web server to serve other requests. But - what if the request must know the result of that ...
0
votes
1answer
666 views

Backing up messages in S3 within a Storm topology

In my project we are trying to build up a KIND-of-a-lambda storm based architecture. The component would be responsible for indexing the site usage events so we expect a quite massive random load. The ...
3
votes
2answers
12k views

Kafka, is it possible to get the message offset after producing?

While using Apache Kafka, is it possible to get the offset of the message that has been produced? From the Kafka documentation page it says: each message in the partition are assigned a unique ...