0

The requirement is to trigger certain tasks (API calls) periodically for each user. But the frequency (time between triggers) is not fixed, the user can change it.

In my existing solution I maintain a table in the DB with userId, freq(seconds), last_run (timestamp)

The service uses a Cron-like library to call a function every second, checks for each entry if (current_time - last_run) > freq and calls the API if true

The solution works at the moment but I am concerned about it scaling, I know I can optimize it but running a query that will only return those entries that satisfy the condition. But is there a better approach?

Problems:

  • Scaling for 10k+ users
  • Race around condition with horizontal scaling
  • Updating last_run for each task could be time consuming (bulk update of all selected works but failed API calls will be marked done)
  • Ensuring time to run all tasks in current tick is less than time between two ticks (currently 1 second, it can be increased to 10 seconds but can it be programmatically constrained?)

Good to have:

  • not a fixed frequency, a cronlike expression for flexibility to choose any schedule

Note:

  • It is built using Nest.js with its own Task Scheduling

  • Most tasks would run every 5minutes (default) unless changed by user

  • Expected to have over 10,000 users each with at least 5 tasks in the next few months

2 Answers 2

1

Here's a possible design that should scale relatively well:

Split the task into a (single threaded) scheduler and multiple workers connected via a task queue. The number of workers can be scaled up and down to adjust for actual load.

The scheduler would be waiting on a timer event for the next run of tasks and on some trigger when the tasks database table changes. How you implement this is a matter of taste, I'd probably use a message queue, too. Since the scheduler's job is only to handle the triggering of tasks via the task queue, it should be fine with 10k+ users. If all of them have scheduled events for the same time, some activations will be slightly delayed, but that's unavoidable. Whenever a batch of tasks have been triggered, the scheduler computes the next wakeup time and sets a timer to fire at that time.

The scheduling rules can be as complex as you want them to be as long as they are computable in reasonable time.

Workers would execute the tasks and can log failure and/or success. It is probably best to keep this separate from the task table which should only be modified when a user changes the schedule for some task.

4
  • This helps, currently the work itself is being doing by different microservices which I can call workers in your terminology. I am calling REST API to trigger it, instead you are suggesting I can pass it as a message on a queue which would make it a lot better without network overhead.
    – shoaib30
    Commented Oct 20, 2021 at 10:59
  • this is how Celery works and it does a good job. The single scheduler thread would a single point of failure and bottle neck, which is okay for now. I just need to be aware of it.
    – shoaib30
    Commented Oct 20, 2021 at 11:05
  • To the last point, if I do not update the table on execution, how will the scheduler know when to run it next as the time to run is not the same for every user.?
    – shoaib30
    Commented Oct 20, 2021 at 11:07
  • My first idea is to update the next wakeup time by the scheduler when it puts the task into the worker queue, not by the worker when it has performed the task. But this indeed something that should be tuned using actual experience. Commented Oct 20, 2021 at 18:23
0

Apache airflow should be a good solution for you, it also has a good UI for monitoring the tasks.

In case you want to implement it own your own.

You need to create a dispatcher that responsible for sending tasks for execution. There should be several independent executors. There could be single dispatcher, and if it cant handle the load you should create several that each one is responsible for different users. (Between the dispatcher and the exexutors you can use a queue like kafka/rabbitmq)

The tasks that need to be executed should be saved in the db, redis, or even in memory in case you dont care about persistence.

You can save it as userid, executionTime, nextExecutionTime, taskStatus

3
  • I am looking more towards a design solution than using another service at the moment. I could look into how AirFlow does it and take some inspiration from it. Thanks
    – shoaib30
    Commented Oct 20, 2021 at 6:35
  • Yes, taking inspiration from AirFlow is a good idea. Ive edited my answer with a proposed design.
    – Bar Hemo
    Commented Oct 20, 2021 at 8:35
  • This is useful, I will do some refactoring and use a queue instead of making HTTP calls.
    – shoaib30
    Commented Oct 20, 2021 at 11:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.