title: Parallel processing in shell scripting, 'pid is not a child of this shell'
body: <p>I have a question about parallel processing in shell scripting. I have a program <code>my
  Program</code>, which I wish to run multiple times, in a loop within a loop. This program is basically this: </p>
  
  <pre><code>MYPATHDIR=`ls $MYPATH`
  for SUBDIRS in $MYPATHDIR; do
    SUBDIR_FILES=`ls $MYPATH/$SUBDIRS`
    for SUBSUBDIRS in $SUBDIR_FILES; do
      find $MYPATH/$SUBDIRS/$SUBSUBDIRS | ./myProgram $MYPATH/$SUBDIRS/outputfile.dat
    done
  done
  </code></pre>
  
  <p>What I wish to do is to take advantage of parallel processing. So I tried this for the middle line to start all the <code>myPrograms</code> at once:</p>
  
  <pre><code>(find $MYPATH/$SUBDIRS/$SUBSUBDIRS | ./myProgram $MYPATH/$SUBDIRS/outputfile.dat &amp;)
  </code></pre>
  
  <p>However, this began all 300 or so calls to <code>myProgram</code> simultaneously, causing RAM issues etc. </p>
  
  <p>What I would like to do is to run each occurrence of <code>myProgram</code> in the inner loop in parallel, but wait for all of these to finish before moving on to the next outer loop iteration. Based on the answers to <a href="https://stackoverflow.com/questions/8038723/wait-for-all-processes-with-a-certain-name-to-finish-in-bash">this question</a>, I tried the following:</p>
  
  <pre><code>for SUBDIRS in $MYPATHDIR; do
    SUBDIR_FILES=`ls $MYPATH/$SUBDIRS`
    for SUBSUBDIRS in $SUBDIR_FILES; do
      (find $MYPATH/$SUBDIRS/$SUBSUBDIRS | ./myProgram $MYPATH/$SUBDIRS/outputfile.dat &amp;)
    done
    wait $(pgrep myProgram)   
  done
  </code></pre>
  
  <p>But I got the following warning/error, repeated multiple times:</p>
  
  <pre><code>./myScript.sh: line 30: wait: pid 1133 is not a child of this shell
  </code></pre>
  
  <p>...and all the <code>myPrograms</code> were started at once, as before.</p>
  
  <p>What am I doing wrong? What can I do to achieve my aims? Thanks.</p>