The requirement is to trigger certain tasks (API calls) periodically for each user. But the frequency (time between triggers) is not fixed, the user can change it.
In my existing solution I maintain a table in the DB with userId
, freq
(seconds), last_run
(timestamp)
The service uses a Cron-like library to call a function every second, checks for each entry if (current_time - last_run) > freq
and calls the API if true
The solution works at the moment but I am concerned about it scaling, I know I can optimize it but running a query that will only return those entries that satisfy the condition. But is there a better approach?
Problems:
- Scaling for 10k+ users
- Race around condition with horizontal scaling
- Updating
last_run
for each task could be time consuming (bulk update of all selected works but failed API calls will be marked done) - Ensuring time to run all tasks in current tick is less than time between two ticks (currently 1 second, it can be increased to 10 seconds but can it be programmatically constrained?)
Good to have:
- not a fixed frequency, a cronlike expression for flexibility to choose any schedule
Note:
It is built using Nest.js with its own Task Scheduling
Most tasks would run every 5minutes (default) unless changed by user
Expected to have over 10,000 users each with at least 5 tasks in the next few months