I'm currently working on refacting an existing project. Basicaly, I need to perform requests to remote services. Current solution is to write tasks to a database and then poll them using cron jobs. This approach creates quite high load on the database. I'm thinking of using a task queue, like Celery for example, to queue requests and provide redundancy in case of services' downtime.
My concern is what if the queue itself goes down? The usage of cron guarantees that task will eventually be processed. And when a task queue is down all tasks are lost. Of course, we can still keep tasks in the database, but it returns us to the problem of polling.
So, what is the right way to go?