Designing a scalable microservice: how to prevent one instance from performing an action if another instance allready did?

Question

As I am moving a part of the monolith app logic to a microservice, I am standing before a problem with scalability.

Currently, the main monolith runs on different instances, and has some scheduled services. Some of those services are pretty straigthforward, but there are some services that are sending emails basing on a result of a database query. The actual solution bases on optimistic locks, the service on the instance that is sending the email first is saving the last notification time on the database, if another instance tries to send the notification to the same user again, this happens:

try {
    final long userId = notificationService.getUserIdToNotify();
        if (nonNull(userId)) {
           mailService.sendNotification(userId));
        }
    } catch (ConcurrencyFailureException e) {
        LOGGER.debug(
            "User has been allready notified by another thread or instance.");
            LOGGER.trace("Optimistic lock", e);
    }

The NotificationService returns an UserId if the user wasn't notified previously, or a null if the user was allready notified.

Doesn't seem to be the best solution, but it somehow works.

I could move the logic to a single microservice, remove the optimistic lock logic, but I have recieved a requirement, which states that the microservice should be prepared for scaling, and therefore I should be aware of those optimistic locks, and design the microservice so the croned services inside it should include logic that will prevent other instances from doing actions more than once (for given period of time when they run).

I am not a microservice expert, but from what I see, there are some design problems rolling at me at dangerous speed.

Should this microservice be scaled? It does some operations on the database on an hourly/daily/weekly basis, and as the logic will be separated from the monolith, it shouldn't affect it... except for performing some operations on the same database which is the monolith using.

And if the service should be scalable, what solution would be best? I was thinking about using Redis to store keys and values of userId and a timestamp, and check those entries before performing actions, to prevent duplicate/unnecessary ones.

I am a little bit stuck now, and something in my lizard brain is saying that there is allready a solution for this, but I just seem to miss it.

Answer 1

I have a suggestion for you how to build put your NotificationService and MailService in the microservice context. I already considered loadbalancing in this suggestion.

If you think about it, you already have two microservices here. One microservice running your scheduled tasks NotificationService and one microservice sending your mails MailService.

A scheduled task, as you use it, is bad decision to scale because it should only run on a given time base. Therefor your scheduled tasks would run in a single instance microservice calling other microservices or executing database queries.

On the other hand your mail microservice is perfect for scaling as its only purpose is to send out mails with a given input. However polling the database on multiple, independant, instances of your mail microservice would lead to multiple notifications, as you already correctly stated because they "don't know anything about each other". So the easiest way here would be to directly notify (not over database query) the mail microservice to notify a given user. That way you don't need to use locks in any way and you can scale your mail microservice as much as you want.

Answer 2

What it needs to be scalable is the sendEmail command. Using messaging for that is best.

It’s common to use a message broker such as RabbitMQ. The microservice responsible for sending emails then consumes the messages from the queue and handles them appropriately.

If you run into a problem of your single instance of email microservice not being enough you can simply scale-out another instance and deploy it instantly.

So the best way will be to replace the need for database polling and prefer streaming and event notifications. If there is new userId to notity just notify the consumer directly.

If you want to keep your scheduling service you have no other choice that deploy this cron-job service alone. This last will be the event producer. But will be best if this cron-job could send event in batch.... like this you will create a more scalable system.

So either going with event-driven system (no more database polling anymore) or keep your cron-job(scheduled service) but try to reimplement the logic to send UserIdToNotifyEvent in batch.

The most important thing here is to split scheduling and send of email logic.