4

We have 2 to 3 dozen microservices that serve our customers. These services are deployed in a Kubernetes cluster, and they're only accessible to the outside world through 3 or 4 API gateways.

We found that sometimes the same data is needed by two or more microservices, so we evaluated a couple of strategies to solve this problem and have implemented a solution in pieces. Like any design, we are not 100% sure if we're using the right approach, and whether we're missing potential pitfalls in the design.

Case 1: When a service of lesser business importance ServiceL needs data from a service of higher business importance ServiceH, then ServiceL calls ServiceH directly to get the necessary data.

Case 2: When a service of lesser business importance ServiceL needs data from many important services ServiceH1, ServiceH2, etc), then ServiceH1, ServiceH2 publish messages with that data to RabbitMQ. The publishing of messages is fire-and-forget, so the services are not blocked. ServiceL subscribes to these messages and stores the data in its own data store. We are okay with the delay in the data becoming available to ServiceL.

Case 3: When a service of higher business importance ServiceH needs data from a less-important service ServiceL, then ServiceL publishes a message with that data to RabbitMQ via a fire-and-forget or blocking mechanism, depending on urgency of syncing the data. ServiceH consumes the message and stores it in its data store. Often the data is needed by ServiceH for reports and summary, and we are okay with the summary not being perfectly up to date at all times.

Case 4: When data is needed by two services and both of them not only read data but also modify it, then we believe the domain identification is wrong in which case we redesign them, often merging these two microservices into one.

Additional Info for Case 2 & 3: Now when we use a messaging framework like RabbitMQ for syncing data across services, over a period of time we observed data is getting out of sync between services. When data gets out of sync, we could see the statistics from RabbitMQ and replay messages, but we believe this brings in unnecessary complexity. We've ended up running jobs once a day to sync the data from the source service to the destination service, where the data is retrieved from the applicable services and not directly from their data stores.

Is this a good approach to sync data between microservices? Are there any pitfalls?

  • Could you clarify Case 4 through more detail? And what do you mean by this in "this brings unnecessary complexity"? – Milan Velebit Apr 18 at 9:34
  • @MilanVelebit, I have edited the question. unnecessary complexity is not for the case 4. It's for Case 2 & 3 where RabbitMq is used. RabbitMQ supports broadcast. While broadcast, what if a message is not delivered to one of the recipient. Should be replay the message? If we replay then some recipients get that message twice. So as per our design. Sending messages is to communicate data, and in this medium of communication there is a probability of failure. And we acknowledge this probability of failure – Nachiappan Kumarappan Apr 18 at 12:45
  • 1
    Having a dedicated process for sync is as I would do it too, however the daily scheduling concerns me. When It comes to synch datasources, the less changes to keep in synch the better. Daily synchs are likely to perform many operations. Have you considered making the synch reactive, so that you can perform the synch based on a operations log length instead of a fixed time rate? – Laiv Apr 19 at 11:03
  • Hey @Laiv, I really like your idea of "the less changes to keep in sync the better". We have a reactive mechanism to sync data. Ideally the data is synced within seconds the data is changed. This is by messages that are sent via RabbitMQ. The mechanism is however now fool proof. This mechanism could fail due to sender not being able to reach RabbitMQ, Due to a RabbitMQ going down, the receiver crashing. (It could event be that the service is newly introduced and it needs to get some data from other services) – Nachiappan Kumarappan Apr 20 at 2:42
  • Does RabbitMQ has HA (high availbility) set up? Additionally, in case of failure, you can Queue messages locally until RMQ is up, then you flush the messages into It. – Laiv Apr 20 at 6:11
0

I'd avoid the separation into "high-business importance" and "low-business importance". Seems like an arbitrary distinction, and one which could change from outside reasons - new business focus, better understanding of the domain etc. - leaving you with some hard-to-change architectural decisions. I'm curious to why this separation exists and what are it's implications on the development side for you?

In general, you should prefer RPCs/direct calls between services whenever possible, and leave queues and other communication mechanisms for special cases - which usually involve performance. And even in those cases, I'd much rather have some backend job doing some eventually consistent update of derived data, than have some services listening on a queue do the same thing in a "streaming" fashion. Many/all of the cases I've encountered in practice of having queues, could have been replaced by a more robust "backend job". It doesn't have to be Hadoop or anything fancy though - just a periodically running binary / cron can do it. There's always a lot of ceremony around queues and services. Many times there's a write to the db and a write to the queue. Which needs special care - tracking of which write was sent, reconciliation jobs for failed writes, retries etc. Not to mention that it makes reasoning about the system harder, backfilling super-hard, local testing grosser etc.

  • There are lot of benefits by separating services as "high business importance" and "low business importance". Let me give you an example to make it relatable. If my business is retailing online, then ProductListingService and OrderingService are very important. However SuggestionService or a jobs that crawl other competing websites are relatively less important. This classification is not arbitrary but comes from a deep understanding of the business. When services are classified like this, it helps me to prioritize bug fixes, resource allocation and scaling of services – Nachiappan Kumarappan Apr 20 at 3:04
  • Could you expand your thought process to say that using non-direct communication between services should only exist for special cases or performance issues ? I feel like it’s the opposite actually especially a micro service architecture as it lets every service be self sufficient and independent of other services. It also makes testing very easy as you only have to test that the right event is sent at the right time. I agree about the high/low importance part though. – Steve Chamaillard Apr 27 at 23:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.