Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [apache-kafka]

Apache Kafka is publish-subscribe messaging rethought as a distributed commit log.

2
votes
0answers
42 views

Data processing pipeline design for processing data

I have a use case for which I need to build a data processing pipeline Customer contact leads data coming from different data sources like csv, data base, api has to be first mapped to a universal ...
1
vote
1answer
23 views

how simulate sequential consumers in Kafka

I have kafka consumer that get data from kafka and do some process if some circumstances meet and send it for another consumer to do it's job. How can i build something like this? One way I think ...
2
votes
0answers
20 views

Data durability guarantees in Kafka

Is it wise to use kafka as the 'source of truth' for mission critical data? The setup is: kafka is the underlying source-of-truth for the data. -querying is done on caches (I.e. Redis, ktables) ...
2
votes
3answers
130 views

How to replace parallel async calls between micro services with event based communication

A service A needs to access services B, C and D to process a request. Currently this is implemented as parallel async calls from A to B, C and D. Once all of them has responded, service A follows some ...
0
votes
0answers
29 views

Send Messages in Spark Streaming

Except Kafka which offers asynchronous communication among nodes in a cluster, how is achieved synchronous communication in Spark-Streaming? For example can I call HTTP request/response somehow?
1
vote
0answers
31 views

Apache Spark guidance - Can it do what I need?

We're working on a large microservices platform for a company in the live events industry. It's made up of containerized RESTful APIs, mostly built in Node, working with Apache Kafka and Cassandra on ...
0
votes
1answer
114 views

Processing only once the same message produced by two producers

If I have two different producers that could produce the same message for a Kafka broker, how can I ensure that only one of the two message occurrences gets processed? Is the only way to have an ...
0
votes
1answer
17 views

Retry mechanism - 3rd party provider

I have the scenario below and what I want to do is to build a retry mechanism to retrieve a status. So, what we have is a payment system and we want to call a 3rd party provided periodically in order ...
2
votes
1answer
106 views

Microservices loose coupling using distributed business logic or centralized?

How do we achieve loose coupling in the below scenario: N microservices (let's call them Callers) needs a logic or business( let's call it Worker) to be executed with different contracts. Although ...
0
votes
0answers
61 views

Apache Kafka as an integration layer

We are creating an application that needs to interacts with a 3rd party API. First we were communicating with this API using HTTP requests, but now someone suggested we use Kafka instead of HTTP ...
0
votes
1answer
105 views

Kafka alternative for small IOT business

we are a small IOT business located in Germany. We have around 1000 installations of our sensor package with 1-4 sensors each. The measurements of these sensors are forwarded to our server and we ...
0
votes
1answer
125 views

Message Broker vs. iPaaS

Does anyone have suggestions/recommendations/guidance they can offer on when to use a message broker like RabbitMQ/Kafka vs. an iPaaS solution like Mulesoft/Jitterbit/Dell Boomi/Workato? Can they ...
0
votes
1answer
66 views

Co-relating messages in Kafka

Context We are building an application that acts like a hub for messages coming in from multiple Producers and it assigns to Consumers based on certain criteria. Need is to have the design highly ...
0
votes
1answer
25 views

Evaluate http reqres vs websockets for geolocation based broadcasts

My use case is that I have a real-time broadcast app where users get feed items based on a geolocation, i.e. a user will receive posts that were broadcasted from within a user-defined radius where (s)...
1
vote
1answer
48 views

What does 'data coverage' mean when talking about ETL processes?

I was watching this talk about ETL's shortcomings and the solutions provided by the Kafka platform but I don't quite understand what the speaker is referring to when she says ETL tools have been ...
1
vote
1answer
94 views

Building a real-time API on a Kafka/Kinesis-centric structure

I am a beginner to the event-driven data architecture using Kafka / Kinesis as the centrepiece (currently Kinesis) and I have some questions regarding how to build a WebSocket API on such a structure. ...
5
votes
1answer
200 views

Is it appropriate to say that RabbitMQ and Apache Kafka solve similar problems?

I have used RabbitMQ but I haven't used Apache Kafka. Is it a similar problem that these products solve, or is there no connection?
1
vote
2answers
982 views

How to build a scalable, containerizable and manageable API with GraphQL, Kafka, Docker, Kubernetes and MongoDB?

I have no doubt regarding what each of the technology does. I have tried all of them as well. What I have doubt with is: In case we use GraphQL, wont it act like a single point of failure if the ...
-1
votes
1answer
410 views

Real time Querying on Real-time Data

I have data flowing from Kafka into MongoDB real-time. This is just raw data and various APIs are served using this data in real time. The APIs respond using aggregation queries. However when data to ...
3
votes
0answers
242 views

Can Event Sourcing and an API Management platform leverage each other?

I'm evaluating the use of Kafka to implement Event Sourcing in a microservices environment that already makes use of an API Management platform. I was wondering if it is common and/or considered a ...
3
votes
1answer
112 views

How to organize exchanges of non critical messages between servers in multiple data centers by using Kafka (or other solution)?

We need to organize a way to exchange messages between servers in multiple data centers. Messages are not critical. We just need to be able to send messages from any servers in any data centers to any ...
1
vote
1answer
1k views

How can I handle multi-producer to particular single-consumer in Kafka?

I'm evaluating the Kafka. It provides some nice features which I want to use, especially durability be default and the access to past messages for new consumers, whenever the consumer wants to reach ...
2
votes
0answers
165 views

Designing clickstream analysis?

I have a application where user purchases/click the certain products. I need to design the click stream analysis here which product got clicked how many number of time, user/geographical detail click ...
7
votes
2answers
4k views

Traditional Message Brokers and Streaming Data

According to the Kafka site: "Kakfa is used for building real-time data pipelines and streaming apps." Searching the internet far and wide, I've found the following generally-accepted definition ...
1
vote
4answers
397 views

What is a “streaming app”, exactly?

I'm trying to get past all the hipster, pie-in-the-sky buzzwording and address a very simple, fundamental question: What is a streaming application? According to the Kafka site "Kafka is used for ...
2
votes
1answer
84 views

Task Assigning at a Distributed System

I want to crawl websites and write output to Solr with my application. My crawler uses Kafka to put crawled website object on a topic and Flume will redirect output to Solr. There are multiple ...
2
votes
1answer
576 views

Event Driven Microservice Race Condition

Say we have 4 services, A through D, which communicate (for the most part) through some sort of asynchronous event-driven system. When a new entity is created in A, B & C receive that event. B ...
1
vote
1answer
245 views

Exploiting Apache Spark Data

and sorry if the question seems a bit naive. I'm currently reading tutorials about Kafka & Spark and there's something I can't figure out : how to exploit / expose the data Spark received. Here'...
5
votes
1answer
2k views

Decoupling microservices with gRPC

I'm setting up a microservices architecture, and am confused about how gRPC can loosely-couple services (compared to a pub-sub message service like Kafka). Doesn't the request go directly to the ...
1
vote
4answers
2k views

Web application Architecture for real time app

Usecase: End user places an order on the company website, the order needs to be pushed to company admin's screen with small latency.The company has multiple branches(identified by compId and branchId ...
1
vote
0answers
463 views

Async stream enrichment with a message queue (e.g. Kafka)

I would like to enrich some streaming data, with one service per enrichment, possibly each service written in a different language and managed/deployed separately. Each service would consume some ...
3
votes
2answers
640 views

Two processes in a single docker container or two services connecting to the same db?

I recently started moving a monolithic application to microservices architecture using docker containers. The general idea of the app is: scraping data -> format the data -> save the data to MySQL -...
1
vote
1answer
145 views

#Apache-flink: Stream processing or Batch processing using Flink

I am tasked with redesigning an existing catalog processor and the requirement goes as below Requirement I have 5 to 10 vendors(each vendor can have multiple stores) who would provide me with 'XML' ...
6
votes
2answers
7k views

Event sourcing, replaying and versioning

I am designing a system that uses Event Sourcing, CQRS and microservices. I am lead to understand this isn't an uncommon pattern. A key feature of the service needs to be the ability to rehydrate/...
2
votes
2answers
3k views

Distributed transactions with Kafka

I need to implement a transaction that spans over loosely coupled (SOA+MOM) components. When a particular event is received, FOO and BAR need to do a transactional operation: that is, both the ...
0
votes
2answers
642 views

Send heavy computation jobs from web server to another process and wait for response

I know it is a common practice to send cpu-bounded heavy tasks to background processes, in order to free a web server to serve other requests. But - what if the request must know the result of that ...
0
votes
1answer
633 views

Backing up messages in S3 within a Storm topology

In my project we are trying to build up a KIND-of-a-lambda storm based architecture. The component would be responsible for indexing the site usage events so we expect a quite massive random load. The ...
3
votes
2answers
9k views

Kafka, is it possible to get the message offset after producing?

While using Apache Kafka, is it possible to get the offset of the message that has been produced? From the Kafka documentation page it says: each message in the partition are assigned a unique ...