I need to design a system that will take jobs (basically calling some web api) and run them at a future time (15 - 45 minutes depending on the job).
The first idea I had was to store the job and the timestamp of when it needs to run in some db and then have workers running each minute looking for jobs to run (select where now() < timestamp limit 1) but then I'd have to store a state (created, in-progress, finished) and then have another cronjob checking for jobs that got stuck in-progress because the worker died. And then some other cronjob deleting finished jobs.
Is there a better way to do this? Some other software build for this kind of thing? The time of execution doesn't need to be exact, just some time after the wait expires, and it should keep the jobs persisted in case of a restart. There's not much of an issue if a job is called twice but it should be prevented if possible.
Edit: the solution doesn't necessarily need to use a SQL database and I would actually prefer if I could avoid a SQL database altogether.