2

A service A needs to access services B, C and D to process a request. Currently this is implemented as parallel async calls from A to B, C and D. Once all of them has responded, service A follows some business logic to create a final result from partial results received from B, C and D. The final result is then written to a DB.

The async parallel communication scheme above has to be replaced by the event based communication shown below

I want to use a message broker (e.g. Kafka) and implement event based communication between the micro services. Upon arrival of a request, A should publish an event with appropriate payload. B, C and D being subscribers, will receive the event and start working on it in parallel. Each of them will publish their respective results with appropriate event codes. My question is, how to combine the partial results from events published by B, C and D?

In the previous approach, A had the luxury to accumulate partial results from each of async calls. But in the event based approach, A only publishes the request_arrived event. So locally accumulating inside the service is not an option.

Edit1: These micro services are deployed as AWS Lambdas. A stays active (and cost us more) while B, C, D are busy. The round-trip delay in B, C, D can be up to 20s. In message based communication A will not have to wait for the other services. That will reduce the infrastructure cost.

Improve this question
2
  • What benefit do you get by using event based communication in this instance?
    – Greg Burghardt
    Nov 15, 2018 at 17:39
  • I'm expecting to reduce the infrastructure cost, please see Edit1 for explanation.
    – Raiyan
    Nov 16, 2018 at 13:32

3 Answers 3

Reset to default
3

With an 'E'

B, C, and D each publish their own completion event. E subscribes to those new events, collecting information from each and storing its state until it recognizes that it has received all the events, potentially in any order. Once it has received them all, it completes its own processing with all the information it needs, possibly publishing its own completion event.

enter image description here

As @kadiii mentioned, see the Saga pattern for handling further complexity, especially for transaction equivalence.

Improve this answer
4

Ideally, you don’t. This sort of approach is fraught with partial failure scenarios. What happens if B and C process, but D dies irrecocably? What happens when you get duplicate data from any of the workers?

In general, you get a new service E which listens for results from BCD, writing the partial results to its own db. Potentially it could be a part of A - it depends on what you’re doing. E could check if it has all of the info and combine it while processing each result. But more likely you would combine the partial results on query from the db or via another service to keep the result processing quick and simple.

Improve this answer
3
  • 1
    Increasingly, I keep hearing about the costs of doing microservices. Are they worth the trouble, in most scenarios?
    – Robert Harvey
    Nov 15, 2018 at 15:50
  • 1
    @RobertHarvey - personally, I tend to think they are for most scenarios (though certainly not all). You need to decompose software to make it vaguely maintainable and flexible. Having a clear separation between parts helps prevent the messy implicit coupling inherent in the worst of software. And the usual cost (someone needs to actually automate deployments, we need to actually figure out how to correlate distributed log messages, we need to figure out how to package shared code) are things that engineers need to do anyways.
    – Telastyn
    Nov 15, 2018 at 17:22
  • 1
    @RobertHarvey it's good that costs are more well-known because that prevents hype driven development. While I personally think MSes tend to be over-applied, it's undoubtedly a viable architecture for many applications. Some critical pieces (such as wide availability and reliability of Kubernetes) are slowly falling into place. Looking at the hype cycle, I'd argue that the industry is currently at the verge of the “plateau of productivity”, whereas some individuals still have to get past their “peak of inflated expectations”.
    – amon
    Nov 17, 2018 at 22:58
1

Are B, C, D real microservices or just functions? If they create meaningful result (e.g. OrderCreated, InvoiceCreated, BillDelivered) in their context I think you should save events created by each of them separately to not block and wait for the slowest. You can than move logic which combine them to event processor and execute only if needed. Although if you need them in some particular order you can use well known pattern for that - Saga.

Improve this answer
1
  • B, C, D are not just functions by any means. They were services owned by three different companies that my company acquired. They all have their own DBs full of user accounts and other resources. Syncing all the accounts is another fun project ;)
    – Raiyan
    Nov 16, 2018 at 13:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.