We are building an application that acts like a hub for messages coming in from multiple Producers and it assigns to Consumers based on certain criteria.
Need is to have the design highly scalable, but at the sametime, core functionality of message-to-interaction co-relation shouldn't be broken.
Main entities in the system:
Producer (Say P1 in this example): Entity that starts an interaction with a Consumer (however producer can't explicitly choose which consumer it can talk to)
Consumer (Say C1 in this example): Entity that serves requests as requested by the Prodcuer.
Interaction (Say I1 in this example) : Is a logical container for messages exchanged between P1 and C1
Message : A information packet to be sent to the other party. Both P1 and C1 can generate a message to be sent to the other party. For this example, we take a scenario of a 3 message interaction in chronological order M1 --> First message in I1 created by P1 M2 --> Second message in I1 created by C1 in response to M1 M3 --> Third message in I1 created by P1 in response to M2
Everytime a Prodcuer is sending a message, the producer has to send interaction ID in the message header so that, the producers messages are routed to the same Consumer who has the context. Example: When P1 is sending M3, he must specify I1 in the payload, so that M3 is correctly routed to C1.
If P1 doesn't provide I1 in M3 message, M3 will be treated like a new message, and hence a new interaction. This will cause the M3 to create a new Interaction Say Ix and land with a different Consumer (Say Cn)
At the entrypoint of the application is a API Gateway which exposes a common message intake format and accepts JSON payloads. Everytime a message (Say M) is received at the API the following co-relation logic is executed (happy path)
IF (M.referenceId == null)
Create a new Interaction //I1
Find a Consumer from a list of consumers who is free. //C1
Assign the interaction to the Consumer //Assign I1 to C1
Else IF (M.referenceId != null)
Check in application database IF an interaction with this ID exists. //check I1.id = M.referenceId
If an interaction is found, associate M with the interaction
Find the Consumer already associated with the interaction and deliver the message.
Here is how the DB would look like
|| Message Id || Created By || Delivered To || Interaction ID ||
|M1 | P1 | C1 | I1 |
| M2 | C1 | P1 | I1 |
| M3 | P1 | C1 | I1 |
Problem Statement: We want to use a queues to be able to scale heavily, but our core operation requires DB access. We have to choose between 2 possible approaches.
Approach#1: Call flow looks like this: API Gateway <--> Kafka Queue <--> NoSQL-DB (or RDBMS for that matter)
Pros: 1. Allows good queueing, Cons: Since the co-relation logic hasn't been executed we can't relate the message and hence can't tell the producer what the interaction ID is
Approach#2: Call flow looks like this: API Gateway <--> NoSQLDB (or RDBMS for that matter)
Pros: Allows the co-relation logic to work and hence we can reliably provide interaction ID to producer.
Cons: Can't utilize the power of Kafka
Question: Is there any approach that can facilitate power of Kafka but at the same time will allow us to do the DB look up logic without causing any slowdown?