Background job in microservices

Question

I have a basic java Dropwizard service within cluster of microservices. It manages user resources with restful api styles. Let's call it user-app Then there is a new operation / api endpoint that takes long time. I decided to have a background jobs (with Quartz) to remove the long running operation from user-app and instead only let the user-app to trigger new quartz job for users/long-operation. This Quartz jobs is called user-scheduler service, a separate service in the cluster. Both user-scheduler and the main user-app service is using user-db database. The original thought of creating user-scheduler is to separate concern when scaling jobs, i.e. we can add/remove user-app without changing user-scheduler.

Now I have concern that these two services are integrating via database, which is a violation of https://microservices.io/patterns/data/database-per-service.html . But is it really though? those two are still part of user-xxx service, and thus they are allowed to talk with database.

So my question is, Do you create separate service for background job? If not, how do you handle scaling for the job?

Answer 1

I think we could provide a more detailed answer having more context (eg. what this job do? which are the responsibilities of user-app?) but in general: yes, that's a very serious antipattern

Imagine we have a users-service which exposes the User resource trough CRUD apis and a userstats-service which is an async job that calculates statistics such as "how many users have the isPremium=true to tell marketing (eg through email reports) how many premium users are registered. This works fine as of now but then two (predictable) things happen

1. the tech organization decides to separate the ownership between the "user management team" (UM) and the "reporting team" (RP)
2. the UM works on a new feature to transition towards a three tiers pricing model: basic, premium, enterprise

What a good UM member would do here is to write a migration that transforms the boolean isPremium into an enumerated column pricingTier. Now the overall system stabilty relies on the fact that someone from UM remembers that shares a contract with the RP and communicates the imminent change, otherwise the userstats-service is gonna be broken, a bug will be opened and a fix provided

One of the main purposes of microservices is to avoid this situation by reducing the contract surface. No shared resources, barely minimum IPC, privileging denormalization. All this burden pays its cost through organizational scalability, which in many cases is more important than system scalability, but this really depends on the company size

But the consequence of this reasoning is not just "don't share the database", but rather ask yourself why to components need to share so many things. 99% of times no splitting is better of a bad splitting

Answer 2

I have a different view on the scheduler you have implemented. A scheduler service/module is largely supposed to be very dumb, you provide it a handler, and this gets invoked at a specific time. The scheduler by itself is unaware of the specifics of the handler and the handler should be implemented in its own business module /service.

With this view, although your scheduler would be a trigger for the module accessing the database, your logic is still in the right place and you don't violate the guideline or accessing db from multiple services.

Answer 3

That's not (yet) an engineering question. You have to provide more (engineering) details like expected requests per seconds, level of isolation (transactions, if any), consistency/availability requirements, whether it is required to persist (checkpoint) running jobs for them to be able to recover faster without recomputed already computed intermediate result, etc. Depending upon that you might get quite different solutions, which require different implementation/maintenance efforts.

Without abovementioned details all the answers/arguments are nothing more but a matter of taste.