Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [apache-kafka]

Apache Kafka is publish-subscribe messaging rethought as a distributed commit log.

0
votes
0answers
14 views

Synchronize Data From Multiple Data Sources

Our team is trying to build a predictive maintenance system whose task is to look at a set of events and predict whether these events depict a set of known anomalies or not. We are at the design ...
0
votes
0answers
12 views

How to manage microservice workflows with Apache Kafka

Currently, we have REST-based services and we would like to make an architectural shift to decoupled microservices with Kafka. However, there are a couple of issues not clear to me. 1- How can we ...
0
votes
1answer
52 views

Kafka cluster workflow working?

I have read lot of articles on google how kafka cluster works. Trying to summarizing my understanding here . Please let me know if any point is not correct here? Producer Side Say Kafka cluster ...
0
votes
2answers
324 views

Maintaining order of events with multiple consumers in a typical pub sub scenario

I am using Kafka. I am developing a simple e-commerce solution. I have a non-scalable catalog admin portal where products, categories, attributes, variants of products, channels, etc are updated. For ...
1
vote
1answer
276 views

Service Bus or Event Stream for Event-Driven Architecture

I'm creating a set of Microservices following Event-Driven Architecture using Event-Carried State Transfer to move state from one service to another. Given the following services; Order service ...
2
votes
0answers
89 views

Data processing pipeline design for processing data

I have a use case for which I need to build a data processing pipeline Customer contact leads data coming from different data sources like csv, data base, api has to be first mapped to a universal ...
1
vote
1answer
30 views

how simulate sequential consumers in Kafka

I have kafka consumer that get data from kafka and do some process if some circumstances meet and send it for another consumer to do it's job. How can i build something like this? One way I think ...
4
votes
0answers
241 views

Data durability guarantees in Kafka

Is it wise to use kafka as the 'source of truth' for mission critical data? The setup is: kafka is the underlying source-of-truth for the data. -querying is done on caches (I.e. Redis, ktables) ...
2
votes
3answers
189 views

How to replace parallel async calls between micro services with event based communication

A service A needs to access services B, C and D to process a request. Currently this is implemented as parallel async calls from A to B, C and D. Once all of them has responded, service A follows some ...
1
vote
0answers
39 views

Apache Spark guidance - Can it do what I need?

We're working on a large microservices platform for a company in the live events industry. It's made up of containerized RESTful APIs, mostly built in Node, working with Apache Kafka and Cassandra on ...
0
votes
1answer
714 views

Processing only once the same message produced by two producers

If I have two different producers that could produce the same message for a Kafka broker, how can I ensure that only one of the two message occurrences gets processed? Is the only way to have an ...
0
votes
1answer
18 views

Retry mechanism - 3rd party provider

I have the scenario below and what I want to do is to build a retry mechanism to retrieve a status. So, what we have is a payment system and we want to call a 3rd party provided periodically in order ...
2
votes
1answer
147 views

Microservices loose coupling using distributed business logic or centralized?

How do we achieve loose coupling in the below scenario: N microservices (let's call them Callers) needs a logic or business( let's call it Worker) to be executed with different contracts. Although ...
1
vote
1answer
271 views

Message Broker vs. iPaaS

Does anyone have suggestions/recommendations/guidance they can offer on when to use a message broker like RabbitMQ/Kafka vs. an iPaaS solution like Mulesoft/Jitterbit/Dell Boomi/Workato? Can they ...
0
votes
1answer
163 views

Co-relating messages in Kafka

Context We are building an application that acts like a hub for messages coming in from multiple Producers and it assigns to Consumers based on certain criteria. Need is to have the design highly ...
0
votes
1answer
34 views

Evaluate http reqres vs websockets for geolocation based broadcasts

My use case is that I have a real-time broadcast app where users get feed items based on a geolocation, i.e. a user will receive posts that were broadcasted from within a user-defined radius where (s)...
1
vote
1answer
49 views

What does 'data coverage' mean when talking about ETL processes?

I was watching this talk about ETL's shortcomings and the solutions provided by the Kafka platform but I don't quite understand what the speaker is referring to when she says ETL tools have been ...
1
vote
1answer
127 views

Building a real-time API on a Kafka/Kinesis-centric structure

I am a beginner to the event-driven data architecture using Kafka / Kinesis as the centrepiece (currently Kinesis) and I have some questions regarding how to build a WebSocket API on such a structure. ...
5
votes
1answer
280 views

Is it appropriate to say that RabbitMQ and Apache Kafka solve similar problems?

I have used RabbitMQ but I haven't used Apache Kafka. Is it a similar problem that these products solve, or is there no connection?
1
vote
2answers
1k views

How to build a scalable, containerizable and manageable API with GraphQL, Kafka, Docker, Kubernetes and MongoDB?

I have no doubt regarding what each of the technology does. I have tried all of them as well. What I have doubt with is: In case we use GraphQL, wont it act like a single point of failure if the ...
-1
votes
1answer
454 views

Real time Querying on Real-time Data

I have data flowing from Kafka into MongoDB real-time. This is just raw data and various APIs are served using this data in real time. The APIs respond using aggregation queries. However when data to ...
4
votes
0answers
320 views

Can Event Sourcing and an API Management platform leverage each other?

I'm evaluating the use of Kafka to implement Event Sourcing in a microservices environment that already makes use of an API Management platform. I was wondering if it is common and/or considered a ...
3
votes
1answer
149 views

How to organize exchanges of non critical messages between servers in multiple data centers by using Kafka (or other solution)?

We need to organize a way to exchange messages between servers in multiple data centers. Messages are not critical. We just need to be able to send messages from any servers in any data centers to any ...
1
vote
1answer
2k views

How can I handle multi-producer to particular single-consumer in Kafka?

I'm evaluating the Kafka. It provides some nice features which I want to use, especially durability be default and the access to past messages for new consumers, whenever the consumer wants to reach ...
2
votes
0answers
245 views

Designing clickstream analysis?

I have a application where user purchases/click the certain products. I need to design the click stream analysis here which product got clicked how many number of time, user/geographical detail click ...
10
votes
2answers
5k views

Traditional Message Brokers and Streaming Data

According to the Kafka site: "Kakfa is used for building real-time data pipelines and streaming apps." Searching the internet far and wide, I've found the following generally-accepted definition ...
1
vote
4answers
496 views

What is a “streaming app”, exactly?

I'm trying to get past all the hipster, pie-in-the-sky buzzwording and address a very simple, fundamental question: What is a streaming application? According to the Kafka site "Kafka is used for ...
2
votes
1answer
85 views

Task Assigning at a Distributed System

I want to crawl websites and write output to Solr with my application. My crawler uses Kafka to put crawled website object on a topic and Flume will redirect output to Solr. There are multiple ...
2
votes
1answer
841 views

Event Driven Microservice Race Condition

Say we have 4 services, A through D, which communicate (for the most part) through some sort of asynchronous event-driven system. When a new entity is created in A, B & C receive that event. B ...
1
vote
1answer
279 views

Exploiting Apache Spark Data

and sorry if the question seems a bit naive. I'm currently reading tutorials about Kafka & Spark and there's something I can't figure out : how to exploit / expose the data Spark received. Here'...
4
votes
3answers
2k views

Decoupling microservices with gRPC

I'm setting up a microservices architecture, and am confused about how gRPC can loosely-couple services (compared to a pub-sub message service like Kafka). Doesn't the request go directly to the ...
1
vote
4answers
3k views

Web application Architecture for real time app

Usecase: End user places an order on the company website, the order needs to be pushed to company admin's screen with small latency.The company has multiple branches(identified by compId and branchId ...
1
vote
0answers
539 views

Async stream enrichment with a message queue (e.g. Kafka)

I would like to enrich some streaming data, with one service per enrichment, possibly each service written in a different language and managed/deployed separately. Each service would consume some ...
3
votes
2answers
689 views

Two processes in a single docker container or two services connecting to the same db?

I recently started moving a monolithic application to microservices architecture using docker containers. The general idea of the app is: scraping data -> format the data -> save the data to MySQL -...
1
vote
1answer
161 views

#Apache-flink: Stream processing or Batch processing using Flink

I am tasked with redesigning an existing catalog processor and the requirement goes as below Requirement I have 5 to 10 vendors(each vendor can have multiple stores) who would provide me with 'XML' ...
8
votes
2answers
8k views

Event sourcing, replaying and versioning

I am designing a system that uses Event Sourcing, CQRS and microservices. I am lead to understand this isn't an uncommon pattern. A key feature of the service needs to be the ability to rehydrate/...
2
votes
2answers
3k views

Distributed transactions with Kafka

I need to implement a transaction that spans over loosely coupled (SOA+MOM) components. When a particular event is received, FOO and BAR need to do a transactional operation: that is, both the ...
0
votes
2answers
695 views

Send heavy computation jobs from web server to another process and wait for response

I know it is a common practice to send cpu-bounded heavy tasks to background processes, in order to free a web server to serve other requests. But - what if the request must know the result of that ...
0
votes
1answer
654 views

Backing up messages in S3 within a Storm topology

In my project we are trying to build up a KIND-of-a-lambda storm based architecture. The component would be responsible for indexing the site usage events so we expect a quite massive random load. The ...
3
votes
2answers
10k views

Kafka, is it possible to get the message offset after producing?

While using Apache Kafka, is it possible to get the offset of the message that has been produced? From the Kafka documentation page it says: each message in the partition are assigned a unique ...