0

I need to design a system that will take jobs (basically calling some web api) and run them at a future time (15 - 45 minutes depending on the job).

The first idea I had was to store the job and the timestamp of when it needs to run in some db and then have workers running each minute looking for jobs to run (select where now() < timestamp limit 1) but then I'd have to store a state (created, in-progress, finished) and then have another cronjob checking for jobs that got stuck in-progress because the worker died. And then some other cronjob deleting finished jobs.

Is there a better way to do this? Some other software build for this kind of thing? The time of execution doesn't need to be exact, just some time after the wait expires, and it should keep the jobs persisted in case of a restart. There's not much of an issue if a job is called twice but it should be prevented if possible.

Edit: the solution doesn't necessarily need to use a SQL database and I would actually prefer if I could avoid a SQL database altogether.

3
  • 1
    Possible duplicate of How to create a timed-event architecture using a SQL database
    – Blrfl
    Commented Jan 30, 2019 at 22:05
  • @Blrfl I don't need the solution to use a sql database and would rather avoid it. Also neither of the answers even mention sql dbs so I think they are in the right track.
    – solarc
    Commented Jan 31, 2019 at 16:41
  • 1
    Don't overlook the value in not having to reinvent the wheels that a database has taken care of. Even SQLite can be rigged up to do this.
    – Blrfl
    Commented Jan 31, 2019 at 17:15

2 Answers 2

1

Look into Hangfire https://www.hangfire.io/ From what you have described it would fit your needs perfectly. You shouldn't have to do anything other than scheduling your jobs as it has built in DB and mechanisms to clean up the jobs and retry them.

1
  • Thank you, it seems to be just what I need, I'll check it out.
    – solarc
    Commented Jan 30, 2019 at 22:03
0

There is existing software available to do this, but it tends to be geared toward distributed microservice systems. Two that I am aware of are Celery and Kubernetes jobs. We use Celery for an internal status tool for doing expensive cache refreshes on a schedule. I've used Kubernetes jobs in the past for scheduling things like automatic backups. They both have a learning curve, but also over time as you need new features, you find their complexity is justified.

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.