Questions tagged [apache-kafka]

Apache Kafka is publish-subscribe messaging rethought as a distributed commit log.

Filter by
Sorted by
Tagged with
-1
votes
0answers
9 views

Design choice: Using kafka for application logging 1 topic vs multiple topics

I'm planning on having multiple applications switch to using kafka as the main place to push logs to asynchronously, where a downstream application will consume those logs and load them somewhere for ...
0
votes
0answers
15 views

Kafka with log compaction best approach

I am publishing entities with entity-id (message-key) + version in kafka. I decided to use log compaction to ensure that atleast 1 version of the entity is available at all times and for that to be ...
0
votes
0answers
16 views

Messaging - Responsibility of Consumer vs Publisher for Active-Active failover

Say there is an external client-DC1/DC2, posting requests to a service in DC1 and DC2 respectively. Now, if DC1 is down, should the external client-DC1 responsible for routing the request message to ...
-4
votes
0answers
12 views

Kafka Consumer - Need to perform batching

Kafka Consumer - Need to perform batching. Use case : How do you perform batching of records at consumer records. For example : Each consumer record contains say single employee record . I want to ...
0
votes
0answers
30 views

How to deal with time-based conditions in an event-driven architecture?

The company I work for has a ticketing platform. The backend system consists of a few applications talking to each other via HTTP and through events in Kafka. A ticket has a sell_start_date field ...
0
votes
0answers
34 views

Designing sane messaging paradigm for concurrent updates

We have a weird use-case that I need to support in my application and needed some opinions on how to design it. At a basic level the application allows users to work on a "list" of "tweets" she cares ...
1
vote
1answer
35 views

Processing messages in parallel with Kafka & Akka

We have data in Kafka which we need to export to multiple destinations. Each message key is to be exported to one destination. A destination can be a REST endpoint, a file, a database etc. Each ...
1
vote
1answer
31 views

Designing Kafka topics for secured event driven job scheduling system

Consider the following Group A Job A { Depends on Job B of Group A Run User -> User1 } Job B { Depends on Job C and Job D of Group A Run User -> User2 } Job C { Depends ...
0
votes
0answers
15 views

Datapipeline integration and analysis

We are working on a project where we need to accept a series of business events related to a set of common users. Those events are produced by various external systems. We need to ingest them, ...
0
votes
0answers
19 views

Difference in AMPS and AMQP

I recently across an acronym AMPS(Asyncronous Message Processing System( by 60EastTechnologies. Since it is not a standard like AMQP(Asynchronous Message Queuing Protocol) or JMS(Java Messaging System)...
0
votes
1answer
133 views

Integrating with real-time data from multiple devices with accelerometer

I am designing a MVP for a simple gamification system for a trampoline park(s). External company is providing bracelets for customers, then collecting this data in their own web app, where they ...
1
vote
2answers
278 views

Microservice architecture pattern for Batch based system

I have been exploring the microservice architecture for the batch-based system. Here is our current setup: Code: We have 5 systems that are internally connected and they pass data from one system ...
2
votes
2answers
100 views

Alternating between Java streams and parallel streams at runtime

This is a question I constantly ask myself when designing a data intensive application: When is it appropriate to use stream() over parallelStream()? Would it make sense to use both? How do I quantify ...
0
votes
0answers
36 views

Deciding on Idempotency Approach on Distirbuted Systems (At Kafka Consumer)

The service that is built works on requests reply method using Kafka consumers. This service communicates with external systems . Whenever an external system requests a message(via KAfka) They had ...
0
votes
1answer
65 views

Will kafka consumers differentiate events on the same topic, or get all events on that topic?

How will a kafka consumer differentiate/behave on the type of events that is stored in a given topic ? Let's say I have a consumer (Consumer1) which is set up to consume from a topic with event of ...
1
vote
0answers
215 views

kafka consumer properties session.timeout.ms vs heartbeat.interval.ms

I am reading Kafka, the definitive guide and I came across the below point for consumers. heartbeat.interval.ms must be lower than session.timeout.ms, and is usually set to one-third of the timeout ...
1
vote
0answers
36 views

Kafka Consumer in Azure

I'm working on a Kafka consumer in C# (using the Confluent C# lib).. and practically all of the documentation/examples online leverage console apps to do the consuming. So, this is fine for logic ...
3
votes
1answer
99 views

How to unit test a service which output is an event with data?

Technologies : NodeJs, Kafka, Mocha The issue : I have a nodeJS service which given a String returns events with a many data. How can I test this using Mocha ? In a simple scenario the service ...
0
votes
0answers
31 views

Efficient Options to do Stream Joins(with Large Time gaps)

Use Case: I need to join two stream sources(say orders(order_id, order_val) and shipments(shipment_id, shipment_val)) based on an id(order_id = shipment_id) and generate a new event shipment_order(id, ...
-1
votes
2answers
70 views

Using Apache Kafka for rare events [closed]

There are multiple microservices which are responsible for storing data dependent from user, e.g. user preferences on some products, user orders, etc. There is a scenario when user is deleted from ...
0
votes
0answers
58 views

Are message identifiers and offsets the same thing in Kafka?

I am reading up on Kafka and encountered the following on one of the tutorials. The producer sends the following to the broker - unique message identifier, timestamp and a binary payload. The ...
1
vote
1answer
119 views

Is Kafka the right choice to decouple my APIs from the service implementation?

I wan't to decouple my exposed API service from their implementation. So I was planning to use KafKa as a mediation layer between my exposed API and the service implementation by going fully ...
0
votes
1answer
100 views

Kafka cluster workflow working?

I have read lot of articles on google how kafka cluster works. Trying to summarizing my understanding here . Please let me know if any point is not correct here? Producer Side Say Kafka cluster ...
0
votes
2answers
2k views

Maintaining order of events with multiple consumers in a typical pub sub scenario

I am using Kafka. I am developing a simple e-commerce solution. I have a non-scalable catalog admin portal where products, categories, attributes, variants of products, channels, etc are updated. For ...
1
vote
1answer
771 views

Service Bus or Event Stream for Event-Driven Architecture

I'm creating a set of Microservices following Event-Driven Architecture using Event-Carried State Transfer to move state from one service to another. Given the following services; Order service ...
3
votes
0answers
130 views

Data processing pipeline design for processing data

I have a use case for which I need to build a data processing pipeline Customer contact leads data coming from different data sources like csv, data base, api has to be first mapped to a universal ...
1
vote
1answer
61 views

how simulate sequential consumers in Kafka

I have kafka consumer that get data from kafka and do some process if some circumstances meet and send it for another consumer to do it's job. How can i build something like this? One way I think ...
9
votes
1answer
951 views

Data durability guarantees in Kafka

Is it wise to use kafka as the 'source of truth' for mission critical data? The setup is: kafka is the underlying source-of-truth for the data. -querying is done on caches (I.e. Redis, ktables) ...
2
votes
3answers
323 views

How to replace parallel async calls between micro services with event based communication

A service A needs to access services B, C and D to process a request. Currently this is implemented as parallel async calls from A to B, C and D. Once all of them has responded, service A follows some ...
0
votes
1answer
2k views

Processing only once the same message produced by two producers

If I have two different producers that could produce the same message for a Kafka broker, how can I ensure that only one of the two message occurrences gets processed? Is the only way to have an ...
0
votes
1answer
25 views

Retry mechanism - 3rd party provider

I have the scenario below and what I want to do is to build a retry mechanism to retrieve a status. So, what we have is a payment system and we want to call a 3rd party provided periodically in order ...
2
votes
1answer
270 views

Microservices loose coupling using distributed business logic or centralized?

How do we achieve loose coupling in the below scenario: N microservices (let's call them Callers) needs a logic or business( let's call it Worker) to be executed with different contracts. Although ...
2
votes
1answer
527 views

Message Broker vs. iPaaS

Does anyone have suggestions/recommendations/guidance they can offer on when to use a message broker like RabbitMQ/Kafka vs. an iPaaS solution like Mulesoft/Jitterbit/Dell Boomi/Workato? Can they ...
0
votes
1answer
257 views

Co-relating messages in Kafka

Context We are building an application that acts like a hub for messages coming in from multiple Producers and it assigns to Consumers based on certain criteria. Need is to have the design highly ...
0
votes
1answer
44 views

Evaluate http reqres vs websockets for geolocation based broadcasts

My use case is that I have a real-time broadcast app where users get feed items based on a geolocation, i.e. a user will receive posts that were broadcasted from within a user-defined radius where (s)...
1
vote
1answer
53 views

What does 'data coverage' mean when talking about ETL processes?

I was watching this talk about ETL's shortcomings and the solutions provided by the Kafka platform but I don't quite understand what the speaker is referring to when she says ETL tools have been ...
1
vote
1answer
193 views

Building a real-time API on a Kafka/Kinesis-centric structure

I am a beginner to the event-driven data architecture using Kafka / Kinesis as the centrepiece (currently Kinesis) and I have some questions regarding how to build a WebSocket API on such a structure. ...
7
votes
1answer
379 views

Is it appropriate to say that RabbitMQ and Apache Kafka solve similar problems?

I have used RabbitMQ but I haven't used Apache Kafka. Is it a similar problem that these products solve, or is there no connection?
1
vote
2answers
2k views

How to build a scalable, containerizable and manageable API with GraphQL, Kafka, Docker, Kubernetes and MongoDB?

I have no doubt regarding what each of the technology does. I have tried all of them as well. What I have doubt with is: In case we use GraphQL, wont it act like a single point of failure if the ...
-1
votes
1answer
507 views

Real time Querying on Real-time Data

I have data flowing from Kafka into MongoDB real-time. This is just raw data and various APIs are served using this data in real time. The APIs respond using aggregation queries. However when data to ...
5
votes
0answers
387 views

Can Event Sourcing and an API Management platform leverage each other?

I'm evaluating the use of Kafka to implement Event Sourcing in a microservices environment that already makes use of an API Management platform. I was wondering if it is common and/or considered a ...
3
votes
1answer
176 views

How to organize exchanges of non critical messages between servers in multiple data centers by using Kafka (or other solution)?

We need to organize a way to exchange messages between servers in multiple data centers. Messages are not critical. We just need to be able to send messages from any servers in any data centers to any ...
1
vote
1answer
3k views

How can I handle multi-producer to particular single-consumer in Kafka?

I'm evaluating the Kafka. It provides some nice features which I want to use, especially durability be default and the access to past messages for new consumers, whenever the consumer wants to reach ...
3
votes
0answers
326 views

Designing clickstream analysis?

I have a application where user purchases/click the certain products. I need to design the click stream analysis here which product got clicked how many number of time, user/geographical detail click ...
14
votes
2answers
7k views

Traditional Message Brokers and Streaming Data

According to the Kafka site: "Kakfa is used for building real-time data pipelines and streaming apps." Searching the internet far and wide, I've found the following generally-accepted definition ...
1
vote
4answers
888 views

What is a “streaming app”, exactly?

I'm trying to get past all the hipster, pie-in-the-sky buzzwording and address a very simple, fundamental question: What is a streaming application? According to the Kafka site "Kafka is used for ...
2
votes
1answer
88 views

Task Assigning at a Distributed System

I want to crawl websites and write output to Solr with my application. My crawler uses Kafka to put crawled website object on a topic and Flume will redirect output to Solr. There are multiple ...
2
votes
1answer
1k views

Event Driven Microservice Race Condition

Say we have 4 services, A through D, which communicate (for the most part) through some sort of asynchronous event-driven system. When a new entity is created in A, B & C receive that event. B ...
1
vote
1answer
313 views

Exploiting Apache Spark Data

and sorry if the question seems a bit naive. I'm currently reading tutorials about Kafka & Spark and there's something I can't figure out : how to exploit / expose the data Spark received. Here'...
4
votes
3answers
5k views

Decoupling microservices with gRPC

I'm setting up a microservices architecture, and am confused about how gRPC can loosely-couple services (compared to a pub-sub message service like Kafka). Doesn't the request go directly to the ...