4

During working on a pet project in microservice event driven architecture, I've faced a problem of delivering responses to a client.

enter image description here

Each microservice (API Gateway etc) has many replicas and has it's own loadbalancer (Kubernetes Service).

Use case:

  1. 20 clients (JS App) sends "Register account request"

  2. 10 clients get connected to "API Gateway 1" and 10 - "API Gateway 2". Gateways hold connection (long polling)

  3. Each API Gateway delegates request processing to Data Processors via Apache Kafka, topic "registerAccount"

  4. Data processors are subscribed to "registerAccount" topic. onMessage() registers account and push new account's ID back to Kafka, topic "newAccountId"

  5. API Gateways receive newAccountId (which one? all of them?)

  6. API Gateways send newAccountId to appropriate client (how can API Gateway decide which client must receive this newAccountId?)

The only resolution I've found is to add client id to each message and then filter messages.

Have anybody experienced such problem?

Improve this question
4
  • 2
    not sure about your framework but rabbit mq supports rpc style calls with a 'reply to queue name' field. each client creates a new private queue and listens for the response
    – Ewan
    Nov 18, 2016 at 12:41
  • please don't cross-post: stackoverflow.com/questions/40676133/… "Cross-posting is frowned upon as it leads to fragmented answers splattered all over the network..."
    – gnat
    Nov 18, 2016 at 12:44
  • 1
    @gnat I've just forgotten to delete old one. Thanks.
    – ovnia
    Nov 18, 2016 at 12:52
  • @Ewan as I've mentioned, its Kafka. As for me, creating new topic each time is not a best solution.
    – ovnia
    Nov 18, 2016 at 12:55

1 Answer 1

Reset to default
3

The way that you effectively load balance a Consumer in Kafka is that you register each of your client nodes/processes into a single Consumer Group.

Consumers label themselves with a consumer group name, and each record published to a topic is delivered to one consumer instance within each subscribing consumer group. Consumer instances can be in separate processes or on separate machines.

If all the consumer instances have the same consumer group, then the records will effectively be load balanced over the consumer instances.

If all the consumer instances have different consumer groups, then each record will be broadcast to all the consumer processes.

[Source] https://kafka.apache.org/intro

This is rather basic Client configuration. There is no concept of a "Message Queue" in Kafka, however you can reliably handle transaction based message driven events in this way.

Improve this answer
5
  • hm...How can this information help me? Let's say each API Gateway is in its own Consumer Group. That means, each message published by Data Processor will arrive to each API Gateway. Indeed, one of these messages will be right one. But it will spam my client with "newAccountId" messages from other registration forms.
    – ovnia
    Nov 18, 2016 at 13:12
  • 1
    @i.ovchynnikov I think I understand now. Your problem is that you need to correlate a response message on a different topic to the publisher of the original request message on a separate topic. You are essentially trying to implement a Request/Response pattern into Kafka. Unfortunately this is not really a use case that Kafka is meant for. That is not to say though that this cannot be handled elegantly either. See stackoverflow.com/questions/35535785/…
    – maple_shaft ♦
    Nov 18, 2016 at 13:38
  • 1
    One of the answers suggests that you have a Request/Response front end to Kafka middleware. newAccount topic messages can be stored in a local DB for a correlation ID such that the original API Gateway that received the request can retrieve the data it needs to formulate the response to the client from the local DB instead of the message directly. It would be inefficient to search partitions for the specific newAccount message that you want. I like this approach.
    – maple_shaft ♦
    Nov 18, 2016 at 13:42
  • Yes, message correlation is exactly what I've mentioned in my first message ("add client id to each message"). But there is another problem - message spamming. Let's say "newAccountId" is a message with "websocket-client-id" inside, so I can identify who was an initiator of registration and a potential notification receiver. Each API gateway instance (each instance in its own CG) will listen for newAccountId's topic and when it receives the message it either forwards msg to client(if it owns socket with such client-id) or just ignore the message. Stronger scaling will cause more spam messages.
    – ovnia
    Nov 18, 2016 at 14:54
  • 1
    @i.ovchynnikov Well a couple things. API Gateways for the newAccount topic should be in a Consumer Group so only one instance of an API Gateway will receive and process the message. Secondly, the message itself is just a notification of new data. The instance that receives and processes the message will store the data somewhere like a DB, but then the original API Gateway will have to fetch the data and build the response from there. It is kind of kludgey and imperfect because you are trying to push Pub/Sub pattern into Req/Resp model.
    – maple_shaft ♦
    Nov 18, 2016 at 15:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.