The requirement is to trigger certain tasks (API calls) periodically for each user. But the frequency (time between triggers) is not fixed, the user can change it.
In my existing solution I maintain a table in the DB with
The service uses a Cron-like library to call a function every second, checks for each entry if
(current_time - last_run) > freq and calls the API if true
The solution works at the moment but I am concerned about it scaling, I know I can optimize it but running a query that will only return those entries that satisfy the condition. But is there a better approach?
- Scaling for 10k+ users
- Race around condition with horizontal scaling
last_runfor each task could be time consuming (bulk update of all selected works but failed API calls will be marked done)
- Ensuring time to run all tasks in current tick is less than time between two ticks (currently 1 second, it can be increased to 10 seconds but can it be programmatically constrained?)
Good to have:
- not a fixed frequency, a cronlike expression for flexibility to choose any schedule