Scheduling jobs to run at a future time

Question

I need to design a system that will take jobs (basically calling some web api) and run them at a future time (15 - 45 minutes depending on the job).

The first idea I had was to store the job and the timestamp of when it needs to run in some db and then have workers running each minute looking for jobs to run (select where now() < timestamp limit 1) but then I'd have to store a state (created, in-progress, finished) and then have another cronjob checking for jobs that got stuck in-progress because the worker died. And then some other cronjob deleting finished jobs.

Is there a better way to do this? Some other software build for this kind of thing? The time of execution doesn't need to be exact, just some time after the wait expires, and it should keep the jobs persisted in case of a restart. There's not much of an issue if a job is called twice but it should be prevented if possible.

Edit: the solution doesn't necessarily need to use a SQL database and I would actually prefer if I could avoid a SQL database altogether.

Answer 1

Look into Hangfire https://www.hangfire.io/ From what you have described it would fit your needs perfectly. You shouldn't have to do anything other than scheduling your jobs as it has built in DB and mechanisms to clean up the jobs and retry them.

Answer 2

There is existing software available to do this, but it tends to be geared toward distributed microservice systems. Two that I am aware of are Celery and Kubernetes jobs. We use Celery for an internal status tool for doing expensive cache refreshes on a schedule. I've used Kubernetes jobs in the past for scheduling things like automatic backups. They both have a learning curve, but also over time as you need new features, you find their complexity is justified.