I'm setting up a microservices architecture, and am confused about how gRPC can loosely-couple services (compared to a pub-sub message service like Kafka). Doesn't the request go directly to the server, and not through a pub/sub system? While gRPC supports asynchronous requests, should I still use pub/sub as a buffer between services to scale them independently?
gRPC can do asynchronous requests, but it is still an RPC pattern and so that is mostly for when you don't care about the answer, not for implementing message (sub/pub) systems.
What I do is, if I need RPC I use gRPC for the communication and ProtocolBuffers to serialize the data, and then when I need messaging I use AMQP, also with ProtocolBuffers.
It makes sense to have the gRPC buffer the data through a messaging system if you need to scale it, have routing between servers, or if you want features like offline messages and also message receipt notification.
My advice is to diagram the communications scenarios on paper, and it should be clear which type of service you need.
Also, note that using gRPC is mostly turn-key and AMQP isn't; generally messaging systems will require you to write a lot of code. If you're not really scaling it very much then it might be an excessive amount of work. Also, I think it is rare that you would want both; the RPC pattern is mostly a subset of the messaging pattern. If you're going to outgrow RPC, use messaging from the start IMO.
gRPC and queues|brokers are not mutually exclusive. It's not like we have to choose one or another. We might choose one, both or none. We could also use an SMTP server as a queue, or a plain text file.
They can complement each other since they satisfy different purposes.
Think in a pipeline where each stage is a process (service).
[Process A] < ----- > [Process B]
Arrows denote cardinality and dependency.
The diagram shows two services that are client and server at the same time. A is client and server of B. And vice-versa. Such a IPC involves services being aware of each other regardless of the communication protocol (HTTP 1/2, web sockets, sockets, etc).
Now, put a queue or broker in-between
[Process A] <----- > [Queue|Broker] <------> [Process B]
Arrows denote cardinality or dependency.
Queues and brokers break the cardinality between services, hence the dependency. They no longer need to be aware of each other. We could even drop services' public interfaces, since they could be no longer needed. No public interface, no coupling.
Queues and brokers are usually designed to be performant/reliable/consistent/scalable/resilient regarding I/O and we can provide our architecture with these features by implementing one or another. For example, gRPC is a transport mechanism for request/response and (non-persistent) streaming use cases. If we can not afford missing messages, we probably will have to look for alternatives or complementary architectural components. Queues and brokers might do the job. Persistence is not the only feature to provide our IPC with. To mention a few
- Message consistency and persistence
- Catching and batching
- Error handling
- Modularity or compartmentality (having dedicated brokers or queues)
- Compatibility with several protocols out-of-the-box
- Traffic traceability
- Traffic monitoring
I found this blog post which dives a little more into the subject.
Worth a mention the Nginx blog post about Microservice and IPC implementations.
Yes the request will go directly to the server.
gRPC allows a developer to generate interfaces for the various services. Clients only need to depend on the interface and not an implementation.
typically interfaces can be designed to be backwards compatible. So when a service changes, new functionality is added to the interface and no functionality removed. Therefore old clients can talk to newer versions of the interface while new clients can access the new features. Once all clients are upgraded then old deprecated parts of the interface can be removed.