5

I have a microservice(Microservice A) which connects to a Client MQ and recieves messages and persists into the database.(Synchronous type MessageA after Message B for the same order). Internally i have n microservices each handling a particular type of message based on some message parameters. All the microservices are independent of each other and can perform processing using the data from Message(Which is persisted in Database -Get RestAPI operation).

However MessageB from the MQ needs to processed only when MessageA has finished its complete processing.(Synchronous). But each message can call N microservices to perform its tasks which can be asynchronous *(Async) in nature.

How can i bring about synchronization between the message processing of parent w.r.t the child (Async tasks). i.e Message A -> 4 tasks(Async microservice) -> calls 4 microservices completes work and processes Message B -4 tasks?

4

When each A sub task finishes, check the state of A. If all sub tasks are complete send a StartB message. If not update state and do next A.

If you B messages are generated first, then you need to put them in holding while you wait for all the As to complete.

You can setup some complex routing to store them in queues, but it might be easier just to pop them in a DB and wait for that StartB message.

edit...

Maybe I should add that if possible you want to avoid this situation, as without a running process to keep track of what sub A tasks have completed, erred or timed out; you have quite a complex error handling problem to solve.

You could add a process which runs through incomplete jobs to check what's happening, but this is exactly the kind of database polling you are trying to avoid with message queues in the first place.

One alternative is to keep a running process, which async calls the sub tasks and waits for a completion message from each. If your tasks are quick this can work well, but it makes this master task synchronous.

Another is to put the B task back on FIFO queue if the start condition is not met. This gives you a chance to error old jobs, but again it means you potentially have services looping over jobs doing nothing but burn cpu time.

  • How about having a messaging Queue such as RabbitMQ where my microservice after persisting will publish the message on the queue (which i guess would contain some complex routing for synchronous processing of records with same ordernumber) and then have a background server process which will read messages from queue and send call the corresponding consumers ?But again error handling is one aspect i am yet not able to solve in this pattern too – Akhil Nov 23 '17 at 5:29
  • you can have error queues to trigger error handling, but at some point you are checking what happened in the other jobs. and that means polling – Ewan Nov 23 '17 at 7:12
0

What about something like a workflow management service?? We have a similar probelem and have a central workflow management service which keeps track of all runing workflow and issues new orders only when alle previous tasks have finished. This makes it really easy to keep track what is happening. A problem could occur if you want to avoid a central managing instance, but I thnk this is mostly justified.

A note to sagas: I really like the idea. But all saga implementations I have seen a limited to a single language (e.g. all services must be writen in c#, as a masstransit.) - which defats the idea of polyglot microservices.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.