0

Overview

Today we have a monolithic service that deals with realtime time data. The monolithic are composed by follow stack: Django + AuroraDB Cluster (Postgres AWS) + Varnish cache and cloudfront at the edge. This service is fed by a third part data provider, which in peak times, provider 5k message/minute, with this, at the provider peak times, we have a requests peas from users, it's about 3 mi req/day. And with this stack, maybe the data take time to arrive to our end user, maybe 30 seconds or more.

This provider, can't fit with our future requirements and we searching about a new one. We find one that match and we need to develop a new application to provide this new data, and refactor our mobile app to match with.

We are thinking about develop this new application with Redis as database, to provide more fast data that we can. But, one of the requirements for the end user is the ability to complex filter data. We have this feature today. But, with redis, isn't possible, given the data structure.

We thinking about to split this problem in two micro services. The first one, with the stack NodeJS + Redis, to provide realtime data, without the filters - this feature needs to provider fast data, because the peak times are from our free users.

The second one, with more "delay" data (Nodejs and Postgres) (avoid lot of writes in peak times, filtering the provider data), with the same entities that first one, but, stored in relational database, that provide the ability of complex filter for our paid end users.

It's a big problem to maintain the same data in two micro services?

Architecture

1
  • Take a look at Celery queue manager and Python multiprocessing. (Although this could be an overkill) May 13 at 17:26

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.