I'm working on a web project that executes CLI commands to perform running tasks to spare users from having to wait since PHP/Apache combo cannot (to my understanding) create a separate process for said long-running processes.

  • Is the this the best approach for running long for running long processes that do not force web users to wait?

  • Is there any time-bombs one needs to look out for when taking this approach (running many CLI processes from PHP/Apache?

  • How would your users get to know the status of their long running task? How would they know when it has finished? What would happen if your server gets flooded with requests for these long running processes? – Bart van Ingen Schenau Jan 21 at 10:33

There are multiple ways to solve something like this:

  • If you're hosting on AWS for example, you might want to look into their SQS and Lambda services. I'm not really familiar with other cloud providers but they probably have similar services of their own
  • A more DIY approach to the queuing-technique is to use Apache Kafka for example (or other similar tools)
  • If your application is written using the Symfony framework, there is a technique in there where you can keep executing stuff AFTER you return the response to the user. In short: it's done by listening to its kernel.terminate event and doing stuff when that event gets triggered
  • If your application is written the Laravel framework: they have quite a nice queueing service which stores the queue in the database/filesystem/redis/whatever and then can be executed from the command-line, therefore also by cronjobs
  • probably 10 other approaches that don't come to mind right now.

Maybe add a few more details about your setup and what you need, that will make it easier to narrow down the best solutions.

Regarding your 2nd bullet point, the things to look out for the most:

  • have a really good way of monitoring those tasks. Since they're in the background, they tend to be more invisible, so you need to break that disadvantage as much as possible. Plan this into your solution from the start
  • have some way of notifying the users when the task is done, if it makes sense. For example: the user uploads an avatar and your background-task needs to resize that image before it's displayed in his profile. Even if notifications don't make sense RIGHT NOW for your use-cases, plan to support them because they might make sense in the future and it will make things a lot easier
  • Decide what to do when a task fails. Do you tell the user? Do you tell the DevOps guys? Do you retry it? If yes, how many times before you give up? Do other tasks depend on its output? Does it leave the database-data in an inconsistent state?
  • If those tasks take a lot of processing power, how do your protect against someone (non)intentionally flooding the system with too many of them?

Some of those things above make sense for your situation, some don't. There are probably many more. Shouldn't be too hard to come up with a more robust list based on your requirements.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.