Best approach to microservice shared databse architecture

Question

I have two microservices, one Flask (python) and one Spring (java), they currently share a database. The Flask microservice handles processing json files (~40mb) for each user (could be 100's or 1000's of users per day) and adds the processed data to a Postgres DB that is shared with the Spring microservice. The Spring microservice is an api that allows users to retrieve data from the DB. The users dont have access to POST, PUT or DELETE endpoints.

I understand that in microservice architecture it is better to have each microservice have its own database. However, these two microservices are heavily linked with the database.

I have thought about introducing a message queue between the services so the Flask service will just send the processed json to a queue and the Spring service will handle the messages from the queue and add the processed data to a database.

I'm worried about overloading the Spring microservice with API requests and adding data to the DB. I understand however that this would be better for scaleability in a Kubernetes system.

Any opinions on how I should handle this? Am I missing an alternative approach that could work better ?

Answer 1

It's less that mSvc's should have their own database and more that each one should be considered a separate application with a singular responsibility. The intention is to allow the owner to only scale up those services that they really need to service their user base and there's really no silver bullets, just trade-offs. And since you're probably doing all of this on Cloud, your costs can change dramatically based on where you put your workload.

Some options here:

• If latency isn't an issue, you could replicate your database with one Primary DB that periodically pushes content to other Child/Secondary DB's.
• You could maybe also use a caching layer (redis?) to support your users read operations since they're not doing any 'PPD' op's.
• You could provide a service layer to act as an intermediary/messaging system between your Spring and Flask mSvc's and your DB.
• Maybe instead of Flask speaking directly to the DB, it handles the raw JSON processing and sends only updates/create operations to an intermediary service which handles the DB processing while your Spring svc connects directly to it. This would allow you to scale up n instances of your Flask service to support your userbase while not bogging down your DB or Spring svc unnecessarily.

Answer 2

I'm worried about overloading the Spring microservice with API requests and adding data to the DB.

If you are worried about performance, delegating more workload to the Spring MS won't make you any good.

I understand that in microservice architecture it is better to have each microservice have its database. However, these two microservices are heavily linked to the database.

When each MS is a whole different application contributing to a greater good. Even when they are, there should be a more powerful reason (other than "theory says...") motivating design decisions like yours.

In this specific case (based on the info we have), both could be "the same service". It happens that, reads and writes are handled apart, and each responsibility has been implemented in different programming languages. That's fine! You have specialized "workers". Python is a great choice for data processing, and Spring is a versatile stack that allows you to deploy web APIs with many different integrations with minimum code.

I have thought about introducing a message queue between the services so the Flask service will just send the processed json to a queue and the Spring service will handle the messages from the queue and add the processed data to a database.

It seems unnecessary complexity. It would be simpler to have a single MS. However, you got 1, composed of 2. That's great because you can scale up/out each of them apart. The more file processing throughput you need, the more Python MS instances you deploy. The more concurrency you have on the fetching side, the more Spring MS you deploy.