0

I'm currently working on refacting an existing project. Basicaly, I need to perform requests to remote services. Current solution is to write tasks to a database and then poll them using cron jobs. This approach creates quite high load on the database. I'm thinking of using a task queue, like Celery for example, to queue requests and provide redundancy in case of services' downtime.

My concern is what if the queue itself goes down? The usage of cron guarantees that task will eventually be processed. And when a task queue is down all tasks are lost. Of course, we can still keep tasks in the database, but it returns us to the problem of polling.

So, what is the right way to go?

1 Answer 1

0

This approach creates quite high load on the database. I'm thinking of using a task queue, like Celery for example, to queue requests and provide redundancy in case of services' downtime.

That's a great idea! You should take the load off the database.

My concern is what if the queue itself goes down?

If that happens, you are sort of hosed unless you have a broker with persistence guarantees like RabbitMQ, Redis Pub/Sub with the append log switched on, etc.. This is no different than if your box running cron decides life is no longer worth living for whatever reason.

The usage of cron guarantees that task will eventually be processed.

Unless the machine shuts down and the hard disk is returned to the ether.

And when a task queue is down all tasks are lost.

Not true in every circumstance, many messaging systems persist to disk.

Of course, we can still keep tasks in the database, but it returns us to the problem of polling.

Don't use your database for this unless you really need to do so. Use a result store, log every request you need to make and reconcile it with what requests you DID make from the result store.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.