I'm working on a web project that executes CLI commands to perform running tasks to spare users from having to wait since PHP/Apache combo cannot (to my understanding) create a separate process for said long-running processes.

  • Is the this the best approach for running long for running long processes that do not force web users to wait?

  • Is there any time-bombs one needs to look out for when taking this approach (running many CLI processes from PHP/Apache?

  • 1
    How would your users get to know the status of their long running task? How would they know when it has finished? What would happen if your server gets flooded with requests for these long running processes? – Bart van Ingen Schenau Jan 21 '19 at 10:33

There are multiple ways to solve something like this:

  • If you're hosting on AWS for example, you might want to look into their SQS and Lambda services. I'm not really familiar with other cloud providers but they probably have similar services of their own
  • A more DIY approach to the queuing-technique is to use Apache Kafka for example (or other similar tools)
  • If your application is written using the Symfony framework, there is a technique in there where you can keep executing stuff AFTER you return the response to the user. In short: it's done by listening to its kernel.terminate event and doing stuff when that event gets triggered
  • If your application is written the Laravel framework: they have quite a nice queueing service which stores the queue in the database/filesystem/redis/whatever and then can be executed from the command-line, therefore also by cronjobs
  • probably 10 other approaches that don't come to mind right now.

Maybe add a few more details about your setup and what you need, that will make it easier to narrow down the best solutions.

Regarding your 2nd bullet point, the things to look out for the most:

  • have a really good way of monitoring those tasks. Since they're in the background, they tend to be more invisible, so you need to break that disadvantage as much as possible. Plan this into your solution from the start
  • have some way of notifying the users when the task is done, if it makes sense. For example: the user uploads an avatar and your background-task needs to resize that image before it's displayed in his profile. Even if notifications don't make sense RIGHT NOW for your use-cases, plan to support them because they might make sense in the future and it will make things a lot easier
  • Decide what to do when a task fails. Do you tell the user? Do you tell the DevOps guys? Do you retry it? If yes, how many times before you give up? Do other tasks depend on its output? Does it leave the database-data in an inconsistent state?
  • If those tasks take a lot of processing power, how do your protect against someone (non)intentionally flooding the system with too many of them?

Some of those things above make sense for your situation, some don't. There are probably many more. Shouldn't be too hard to come up with a more robust list based on your requirements.

| improve this answer | |

Assuming a Linux operating system (on the web server), you might consider installing the batch and/or at utilities and use them from your web application (BTW, you could consider better tools than PHP for it, e.g. ocsigen). You definitely should master Linux system programming techniques since you are using several processes simultaneously (the web server and your CLI running command).

You could be interested in FastCGI (or use HTTP proxy techniques on your web server, with perhaps some HTTP server library like libonion used in your own server program with an event loop monitoring long-lasting tasks) ....

The important issue is how to notify your web user of the completion of the long lasting Linux process started on the web server. Maybe inter-process communication techniques (see poll(2), pipe(7), unix(7), ....) are relevant (between long lasting programs and web service code). Read also ALP. Of course you want to protect your web service against DoS attacks and you need to care about job control (so be at least aware of setrlimit(2), time(7), signal(7), fork(2), execve(2), wait4(2), dup2(2) etc...).

You could be interested in inotify(7) facilities. The long lasting job could run in some batch script ending with e.g. an echo done > somefile to create a file, and you would use inotify facilities for synchronization purposes in your web server.

Assuming recent graphical browsers used for your web application, you might consider using Websockets and AJAX techniques: on the web server side, you would use polling techniques or some event loop, and when the job is completed you'll asynchronously send on the websocket some JSON message, with some AJAX javascript code in the browser to handle it and display something nice when that happens (so passive polling in the browser too...)

PS. See also this question. Several (but not all, and not most) points there are relevant for your needs.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.