I am designing a somewhat complex flow in publish subscribe pattern using AWS SNS, SQS , Microservice and Lambda functions. To break the whole flow into multiple logical & smaller sub-flows I am trying to accomplish that through publishing messages in different steps of the flow. But it seems the same microservice only has to consume those messages. In other words the producer and the consumer of the messages is same Mx. Is it a good pattern or should I create separate lambda functions to do each of the steps? Please suggest. A diagram of the pattern I am thinking is given below.
Sure, there can be many valid reasons for doing this depending on the business process of the application.
One simple example for a stock management application, where you could have the following flow:
Product Microservice receives a command to remove a certain unit of a a product from stock. This command itself is successful if the units are available, but the microservice is also responsible for determining if more of that product needs to be ordered from suppliers. In that case, the command would also trigger an event (message) to be handled by the same microservice, to perform that task in a separate process.
A more general way to think about it is that you have an action which can be considered consistent & done, even though there are post-processing steps which should still be done. These post-processing steps can be triggered as messages to self.
There are some trade-offs to consider, of course. Here are just some of them:
When you scale it out, the intermediate/post-processing messages can be handled by any instance of the microservice, making more efficient use of the resources when compared to a simple single or even multithreaded processing inside a single instance.
The system needs to resilient to the fact that there will be eventual consistency between the changes triggered by each message in the overall process. I.e., it should not be strange/surprising for the users of that system, if the UI says the system did something, but there are still out-of-date information presented in other parts of the system.
You are effectively losing encapsulation over internal operations of your system, and with that, each operation you send via messages has an additional cost to you in the maintenance of your codebase and operation of the service, in case you need to support any of the below qualities:
No downtime deployment: to support that, you need to always ensure that a newly deployed version of your microservice is still able to handle messages published by the previous version (as would happen in a blue-green deployment approach)
Replayability of the messages: to support that, you will always need to keep backwards compatibility of previous versions of a message, or at least as far back as you allow them to be replayed (if you have an event store, for example)
We did this in one of our services. It felt cleaner because everything was going through the same interface instead of having a separate 'internal-to-the-service' interface that behaved slightly differently.
However, we eventually decided to change it. The problem was at scale, the queue would get to a good size. Our follow-up event would get queued behind all the older incoming events, and that caused some weird ordering issues. It would process the first part of 100+ events before processing the second part of the first event. In our particular case that was problematic. It may be fine in your case, or you may have other ways to mitigate that problem, but it's something to be aware of.