Co-relating messages in Kafka

Question

Context

We are building an application that acts like a hub for messages coming in from multiple Producers and it assigns to Consumers based on certain criteria.

Need is to have the design highly scalable, but at the sametime, core functionality of message-to-interaction co-relation shouldn't be broken.

Main entities in the system:

Producer (Say P1 in this example): Entity that starts an interaction with a Consumer (however producer can't explicitly choose which consumer it can talk to)

Consumer (Say C1 in this example): Entity that serves requests as requested by the Prodcuer.

Interaction (Say I1 in this example) : Is a logical container for messages exchanged between P1 and C1

Message : A information packet to be sent to the other party. Both P1 and C1 can generate a message to be sent to the other party. For this example, we take a scenario of a 3 message interaction in chronological order M1 --> First message in I1 created by P1 M2 --> Second message in I1 created by C1 in response to M1 M3 --> Third message in I1 created by P1 in response to M2

Everytime a Prodcuer is sending a message, the producer has to send interaction ID in the message header so that, the producers messages are routed to the same Consumer who has the context. Example: When P1 is sending M3, he must specify I1 in the payload, so that M3 is correctly routed to C1.

If P1 doesn't provide I1 in M3 message, M3 will be treated like a new message, and hence a new interaction. This will cause the M3 to create a new Interaction Say Ix and land with a different Consumer (Say Cn)

At the entrypoint of the application is a API Gateway which exposes a common message intake format and accepts JSON payloads. Everytime a message (Say M) is received at the API the following co-relation logic is executed (happy path)

IF (M.referenceId == null)
{
    Create a new Interaction //I1
    Find a Consumer from a list of consumers who is free. //C1
    Assign the interaction to the Consumer //Assign I1 to C1
}
Else IF (M.referenceId != null)
{
    Check in application database IF an interaction with this ID exists. //check I1.id = M.referenceId
    If an interaction is found, associate M with the interaction
    Find the Consumer already associated with the interaction and deliver the message.
}

Here is how the DB would look like

---

|| Message Id || Created By || Delivered To || Interaction ID ||

|M1 | P1 | C1 | I1 |

| M2 | C1 | P1 | I1 |

| M3 | P1 | C1 | I1 |

Problem Statement: We want to use a queues to be able to scale heavily, but our core operation requires DB access. We have to choose between 2 possible approaches.

Approach#1: Call flow looks like this: API Gateway <--> Kafka Queue <--> NoSQL-DB (or RDBMS for that matter)

Pros: 1. Allows good queueing, Cons: Since the co-relation logic hasn't been executed we can't relate the message and hence can't tell the producer what the interaction ID is

Approach#2: Call flow looks like this: API Gateway <--> NoSQLDB (or RDBMS for that matter)

Pros: Allows the co-relation logic to work and hence we can reliably provide interaction ID to producer.

Cons: Can't utilize the power of Kafka

Question: Is there any approach that can facilitate power of Kafka but at the same time will allow us to do the DB look up logic without causing any slowdown?

Answer 1

I think that Kafka is exactly the wrong choice in this situation.

Kafka works best when a topic is read by consumers that are able to take a chunk of records, process them, and record the offsets.

In your case, you have consumers that will process single messages. And while Kafka can do this, you're paying a high IO penalty relative to throughput.

Moreover, you have a situation where you have an undifferentiated queue that can be read by any consumer, and pairs of differentiated queues that represent communications between producers and a specific consumer. To do this sanely, you'll need to create a separate topic for each interaction (actually, two of them, for bidirectional communication). This all but eliminates the benefit of sequential writes that I consider the true "power" of Kafka.

Instead, your application sounds like a good use of an AMQP-style queue, where you can use addressing to route messages. Or alternatively, some form of persistent TCP connection with a load balancer to establish the interaction (nginx will do this). If your producers are actually web clients, then web sockets seems like a good idea, since much of the implementation already exists.