0

Overview

Today we have a monolithic service that deals with realtime time data. The monolithic are composed by follow stack: Django + AuroraDB Cluster (Postgres AWS) + Varnish cache and cloudfront at the edge. This service is fed by a third part data provider, which in peak times, provider 5k message/minute, with this, at the provider peak times, we have a requests peas from users, it's about 3 mi req/day. And with this stack, maybe the data take time to arrive to our end user, maybe 30 seconds or more.

This provider, can't fit with our future requirements and we searching about a new one. We find one that match and we need to develop a new application to provide this new data, and refactor our mobile app to match with.

We are thinking about develop this new application with Redis as database, to provide more fast data that we can. But, one of the requirements for the end user is the ability to complex filter data. We have this feature today. But, with redis, isn't possible, given the data structure.

We thinking about to split this problem in two micro services. The first one, with the stack NodeJS + Redis, to provide realtime data, without the filters - this feature needs to provider fast data, because the peak times are from our free users.

The second one, with more "delay" data (Nodejs and Postgres) (avoid lot of writes in peak times, filtering the provider data), with the same entities that first one, but, stored in relational database, that provide the ability of complex filter for our paid end users.

It's a big problem to maintain the same data in two micro services?

Architecture

1
  • Take a look at Celery queue manager and Python multiprocessing. (Although this could be an overkill) May 13, 2022 at 17:26

1 Answer 1

1

We had the same doubt, I think this info could help 6 months later too, I found this information useful in some cases, because generally the software catalog is a 'centralized tool' that keeps track of ownership and metadata for all the software built by an organization, so about your case it could serve using 'Event-Driven' Microservices with Python and Apache Kafka.

And you could say why?

  1. Can 2 Microservices connect to same database?

In the shared-database-per-service pattern, the same database is shared by several microservices. You need to carefully assess the application architecture before adopting this pattern, and make sure that you avoid hot tables (single tables that are shared among multiple microservices).

  1. Should each microservice have its own Database?

Each microservice must own its domain data and logic. Just as a full application owns its logic and data, so must each microservice own its logic and data under an autonomous lifecycle, with independent deployment per microservice.

  1. Do we need separate database for microservices?

As you described it above, each microservice needs to own it's DATA, which could be held within a dedicated database, within a dedicated schema (within a database), or even a set of dedicated tables (within schema within a database).

  1. Can we deploy multiple microservices on same server?

One container per VM would be overkill to take away advantages of containers.

  1. How to maintaining data consistency between microservices?

A basic principle of microservices is that each service manages its own data. Two services should not share a data store, so to extend our systems by adding new applications that consume the same events without affecting the existing flow.

Possible solution to simplify the process would be:

Using events with a platform like Apache Kafka® can dramatically reduce the coupling between our applications and make it easier to keep our microservices from becoming a distributed monolith.

Through asynchronous communication (for example, message-based communication) across internal microservices, use retries with exponential backoff, work around network timeouts, use the Circuit Breaker pattern, provide fallbacks, limit the number of queued requests.

Events in Kafka are durable, we can also replay them at a later time or use them to produce data products that can be of value to other parts of the organization.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.