1

We have an app that is using a fairly simple stack (Linux, PHP, Oracle, Shell Scripts, etc).

We have a series of scripts that need to be executed:

  • /scr/app1/start.sh
  • /scr/app2/start.php
  • /scr/app3/kickoff.py
  • /scr/app4/test.sh

The dependency rules are as follows:

  • app1 must be run first
  • app2 then must be run next
  • app3 and app4 can be run in parallel

That being said, I am looking to capture the following from ALL scripts:

  • exit codes (used to determine success of failure / breakage of chain)
  • standard out and/or standard error

The solution does not necessarily have to be written in PHP, it just has to work. Node.js / io.js is part of our build stack so we are comfortable building there.

Basically the parallelism is the real challenge that I see. What is the best approach to this problem while satisfying all of the requirements?

5
  • You get parallelism from the OS by spawning separate processes for each task you want to run in parallel. You get serial execution by running one task and then monitoring its completion before you start the next task. The specifics of how to do this depend entirely upon which development environment you choose to write the controlling program in.
    – jfriend00
    Commented Mar 4, 2015 at 1:21
  • "Basically the parallelism is the real challenge that I see." No, you get that for free from the OS. The hard part is probably building the dependency graph, unless you only ever need to deal with this specific case.
    – Doval
    Commented Mar 4, 2015 at 1:26
  • I should've mentioned I'm a total newbie in the parallelism arena. It looks like PHP has this php.net/manual/en/function.pcntl-fork.php available. Is that the right direction to go in? Commented Mar 4, 2015 at 1:31
  • Probably not. You want a function to run shell commands. It shouldn't be hard to Google.
    – Doval
    Commented Mar 4, 2015 at 2:07
  • The function to run shell commands is not the issue.. there are multiple available in PHP such as shell_exec, system, proc_open, etc. I am looking for a way to thread / fork multiple processes. See my answer below. Commented Mar 5, 2015 at 2:30

4 Answers 4

1

You can write a shell script to do this.

  1. Run app1 and redirect the stdout and stderr to a file. Check the exit code, if fail, exit.
  2. Same as above.
  3. for running app3 and app4 parallely, you can run it in background. To capture the output and return codes, you should take a look into bash coproc (http://wiki.bash-hackers.org/syntax/keywords/coproc). Redirect the outputs to separate files, so they don't get messed up, later you can concat it if you wish.
1
  • coproc looks interesting, thanks for the advice. We have Bash 3.2.51 on our boxes, so unfortunately this solution does not work. We can deploy Bash 4 but I'd rather not go that route yet. Commented Mar 5, 2015 at 2:27
1

Probably not the best way to do it.. But if the dependency tree is a DAG, we can use a topological sort to find an ordering that will make sense, and then we can use bash to call the scripts.

For example, If the dependency tree is like this

enter image description here

We can write a script like this.. As i found from this link, GNU tail can wait for non child processes. We can make use of that.

   #! /bin/bash
   function pswait(){
           tail --pid=$1 -f /dev/null
   }

   B & PID_B=$!
   E & PID_E=$!
   G & PID_G=$!
   ( pswait $PID_B; A ) &
   ( pswait $PID_B; C ) &
   ( pswait $PID_E; pswait $PID_G; F ) & PID_F=$!
   ( pswait $PID_B; pswait $PID_F; D ) &


         

I am sure this can be automated if the need arises using the topological sort.. Or might have already been done somewhere.

Since all of them are in standard bash, checking for error codes and manipulating the stderr and stdout shouldnt be an issue.

One issue with this approach is, unlike wait, the approach used by tail is polling to see if the process has exited. So in a long dependency graph, all those processes will keep polling until it gets the chance to run.

0

So the "PHP" way to do it is via the pcntl (Process Control) extension. This extension is NOT supported on Windows, so it must be used in *nix environments only.

That being said, pcntl_fork is where the magic lies. While there is a third-party extension for threading (vs. forking) in PHP, it is not officially supported, not well tested and not being actively developed. I.e., don't use it.

Forking seems to have it's own set of problems (you can google forking vs. threading), but all around, it is the easiest and most simple / maintainable way to do parallel computing.

I'm writing a library that takes advantage of symfony/process to run processes and get the buffer. Previously, I used a StackOverflow answer to safely asynchronously run a process without any blocking in PHP. But I will be replacing it with symfony/process and editing this answer with a GitHub link when I publish it.

As for the dependency chain, the library I'm working on takes a json object (to keep things as simple as can be), which describes the sequence of processes to execute, including which to run in parallel.

0

I'd use node.js to initiate each of the tasks that need to be done, and I would manage your parallel sequencing of things with a promise library like Q.

With Q, you can basically say:

1) Do all these things at the same time and let me know when you are done with all of them

2) Then, now that you've finished all those things, do something else either by itself or at the same time as some other things (my choice)

You can keep chaining like this, and each "chain-link's stuff" gets done roughly in parallel, but nothing gets started until all prerequisite tasks have completed.

Scroll down this page for examples: https://github.com/kriskowal/q

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.