I wan't to decouple my exposed API service from their implementation. So I was planning to use KafKa as a mediation layer between my exposed API and the service implementation by going fully asynchronous and using a request/reply pattern with kafka. Here a schema of what I am planning to do :

enter image description here

I think that with this architecture I could easily :

  • separate the entry point in the system from my service implementation
  • easily allow scale up of the solution
  • allow service to go offline for maintenance/upgrade

But I am also wondering if this is the right pattern to choose because I never seen it anywhere used before? Do you see any possible drawback with my architecture?

  • I have seen a similar architecture where Kafka was meant to deal with high traffic of requests addressed to a reporting service (somewhat your Service A, B, C). What you called #1 and #2 was an API gateway built in NodeJS dispatching messages to Kafka. However, it was one-way communication, the client never got anything different than a 202 Accepted. So, regarding the pattern request/replay, I'm sorry to say I don't know how it will behave or how to treat the async IPC – Laiv Jul 9 '19 at 10:56

A few things to consider in your architecture:

separate the entry point in the system from my service implementation

It depends if you want to expose capabilities of your services and create api gateway or you just want to replace direct communication with messages, increasing availability and decrease latency of your system. Think twice if you will need some data/business logic before creating the message published to Kafka.

easily allow scale up of the solution

Although event-driven approach may increase your throughput it is not increasing capability of your system to scale up. Splitting wisely your services and creating stateless system do. E.g. consider, in your solution, how easy it will be to read messages from Kafka from many nodes. This may help you to realize if you can easily scale or not your entire solution. Your system is most likely not only about gathering request but also about processing.

allow service to go offline for maintenance/upgrade

It also depends. Your business case can either allow it or not. You will need to properly handle different scenarios, especially when some operation fails. You will also need to handle messages queued while your workers stop reading messages. There are simpler solutions to meet requirement of no downtime maintenance/upgrade than that one.

To sum up, you need to think about the details and probably consider lower level of abstraction to see potential difficulties.

  • For your first remark : I do not understand. I effectively plan to create an api gateway (to manage multitenant authentication, security ) and possibly let my service evolve. – Dypso Jul 9 '19 at 17:37
  • The primary responsibility of an api gateway is to expose capabilities of your internal services adjusted to end user (meaning you can have e.g. different api gateway for mobile and different for frontend app). There are although several challenges while creating such an 'aggregated' api, avoiding moving business rules to the gateway to name one. It is also common to add extra responsibilities like authentication. Be careful here as well as you may end up creating side communication channels to retrieve data from your core services. – Tomasz Maciejewski Jul 10 '19 at 12:22
  • Putting queue in between makes it even more tricky as opposed to api gateway queues tend to operate on single events (not aggregated). It is also often a case that some operations needs to be performed in order (in different services) or requires transaction. not saying it is not a good idea, just requires extra attention as any decoupling creates elements less connected and harder to model the flow. I really recommend you to add more details to your drawing so that you can see the real flow and the real difficulties. – Tomasz Maciejewski Jul 10 '19 at 12:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.