Considerations for delegating long running to CLI from PHP/Apache

Question

I'm working on a web project that executes CLI commands to perform running tasks to spare users from having to wait since PHP/Apache combo cannot (to my understanding) create a separate process for said long-running processes.

• Is the this the best approach for running long for running long processes that do not force web users to wait?

• Is there any time-bombs one needs to look out for when taking this approach (running many CLI processes from PHP/Apache?

Answer 1

There are multiple ways to solve something like this:

• If you're hosting on AWS for example, you might want to look into their SQS and Lambda services. I'm not really familiar with other cloud providers but they probably have similar services of their own
• A more DIY approach to the queuing-technique is to use Apache Kafka for example (or other similar tools)
• If your application is written using the Symfony framework, there is a technique in there where you can keep executing stuff AFTER you return the response to the user. In short: it's done by listening to its kernel.terminate event and doing stuff when that event gets triggered
• If your application is written the Laravel framework: they have quite a nice queueing service which stores the queue in the database/filesystem/redis/whatever and then can be executed from the command-line, therefore also by cronjobs
• probably 10 other approaches that don't come to mind right now.

Maybe add a few more details about your setup and what you need, that will make it easier to narrow down the best solutions.

Regarding your 2nd bullet point, the things to look out for the most:

• have a really good way of monitoring those tasks. Since they're in the background, they tend to be more invisible, so you need to break that disadvantage as much as possible. Plan this into your solution from the start
• have some way of notifying the users when the task is done, if it makes sense. For example: the user uploads an avatar and your background-task needs to resize that image before it's displayed in his profile. Even if notifications don't make sense RIGHT NOW for your use-cases, plan to support them because they might make sense in the future and it will make things a lot easier
• Decide what to do when a task fails. Do you tell the user? Do you tell the DevOps guys? Do you retry it? If yes, how many times before you give up? Do other tasks depend on its output? Does it leave the database-data in an inconsistent state?
• If those tasks take a lot of processing power, how do your protect against someone (non)intentionally flooding the system with too many of them?

Some of those things above make sense for your situation, some don't. There are probably many more. Shouldn't be too hard to come up with a more robust list based on your requirements.