We have an app that is using a fairly simple stack (Linux, PHP, Oracle, Shell Scripts, etc).

We have a series of scripts that need to be executed:

  • /scr/app1/start.sh
  • /scr/app2/start.php
  • /scr/app3/kickoff.py
  • /scr/app4/test.sh

The dependency rules are as follows:

  • app1 must be run first
  • app2 then must be run next
  • app3 and app4 can be run in parallel

That being said, I am looking to capture the following from ALL scripts:

  • exit codes (used to determine success of failure / breakage of chain)
  • standard out and/or standard error

The solution does not necessarily have to be written in PHP, it just has to work. Node.js / io.js is part of our build stack so we are comfortable building there.

Basically the parallelism is the real challenge that I see. What is the best approach to this problem while satisfying all of the requirements?

  • You get parallelism from the OS by spawning separate processes for each task you want to run in parallel. You get serial execution by running one task and then monitoring its completion before you start the next task. The specifics of how to do this depend entirely upon which development environment you choose to write the controlling program in. – jfriend00 Mar 4 '15 at 1:21
  • "Basically the parallelism is the real challenge that I see." No, you get that for free from the OS. The hard part is probably building the dependency graph, unless you only ever need to deal with this specific case. – Doval Mar 4 '15 at 1:26
  • I should've mentioned I'm a total newbie in the parallelism arena. It looks like PHP has this php.net/manual/en/function.pcntl-fork.php available. Is that the right direction to go in? – Avindra Goolcharan Mar 4 '15 at 1:31
  • Probably not. You want a function to run shell commands. It shouldn't be hard to Google. – Doval Mar 4 '15 at 2:07
  • The function to run shell commands is not the issue.. there are multiple available in PHP such as shell_exec, system, proc_open, etc. I am looking for a way to thread / fork multiple processes. See my answer below. – Avindra Goolcharan Mar 5 '15 at 2:30

You can write a shell script to do this.

  1. Run app1 and redirect the stdout and stderr to a file. Check the exit code, if fail, exit.
  2. Same as above.
  3. for running app3 and app4 parallely, you can run it in background. To capture the output and return codes, you should take a look into bash coproc (http://wiki.bash-hackers.org/syntax/keywords/coproc). Redirect the outputs to separate files, so they don't get messed up, later you can concat it if you wish.
| improve this answer | |
  • coproc looks interesting, thanks for the advice. We have Bash 3.2.51 on our boxes, so unfortunately this solution does not work. We can deploy Bash 4 but I'd rather not go that route yet. – Avindra Goolcharan Mar 5 '15 at 2:27

So the "PHP" way to do it is via the pcntl (Process Control) extension. This extension is NOT supported on Windows, so it must be used in *nix environments only.

That being said, pcntl_fork is where the magic lies. While there is a third-party extension for threading (vs. forking) in PHP, it is not officially supported, not well tested and not being actively developed. I.e., don't use it.

Forking seems to have it's own set of problems (you can google forking vs. threading), but all around, it is the easiest and most simple / maintainable way to do parallel computing.

I'm writing a library that takes advantage of symfony/process to run processes and get the buffer. Previously, I used a StackOverflow answer to safely asynchronously run a process without any blocking in PHP. But I will be replacing it with symfony/process and editing this answer with a GitHub link when I publish it.

As for the dependency chain, the library I'm working on takes a json object (to keep things as simple as can be), which describes the sequence of processes to execute, including which to run in parallel.

| improve this answer | |

I'd use node.js to initiate each of the tasks that need to be done, and I would manage your parallel sequencing of things with a promise library like Q.

With Q, you can basically say:

1) Do all these things at the same time and let me know when you are done with all of them

2) Then, now that you've finished all those things, do something else either by itself or at the same time as some other things (my choice)

You can keep chaining like this, and each "chain-link's stuff" gets done roughly in parallel, but nothing gets started until all prerequisite tasks have completed.

Scroll down this page for examples: https://github.com/kriskowal/q

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.