Answers.Body: <p>First of all <em>never</em> do <code>for foo in $(find ...)</code>. That is very bad practice as it can't handle even slightly odd file names like those with a space. See <a href="https://mywiki.wooledge.org/BashPitfalls#for_f_in_.24.28ls_.2A.mp3.29" rel="nofollow noreferrer">bash pitfall #1</a>.</p>
  <p>Next, you don't need <code>pwd</code>, just run <code>find -type f -name &quot;*.fa&quot;</code> which will search in the current directory by default and unlike <code>find `pwd`... </code>, won't break if your path contains whitespace.</p>
  <p>That's just general best practice, the real issue here is that you are probably running the script like this:</p>
  <pre><code>sh /path/to/script.sh
  </code></pre>
  <p>Right? And you are using a Debian-based system like Ubuntu, right? On such systems, <code>/bin/sh</code> is actually not bash but a very basic POSIX compatible shell called <code>dash</code> and <code>dash</code> doesn't understand the <code>(( i++%waitevery==0 ))</code> syntax.</p>
  <p>Since you have a shebang line (<code>#!/bin/bash</code>), you can simply make your script executable (<code>chmod a+x /path/to/script.sh</code>) and then run the script directly like this:</p>
  <pre><code>/path/to/script.sh
  </code></pre>
  <p>Or, alternatively, run it explicitly with <code>bash</code> instead of <code>sh</code>:</p>
  <pre><code>bash /path/to/script.sh
  </code></pre>
  <p>Either should work and if your file names and paths are sane, the issues I mentioned above won't be relevant. But to keep things nice and clean, I would also change the script to (note that this assumes you are using Linux and GNU <code>find</code>):</p>
  <pre class="lang-sh prettyprint-override"><code>#!/bin/bash
  i=1
  waitevery=30
  
  mkdir -p out
  
  find -type f -name &quot;*.fa&quot; -print0 |
    while IFS= read -r -d '' j; do 
      echo &quot;Iteration: <span class="math-container">$i; File: $</span>j&quot;
      filename=<span class="math-container">$(basename "$</span>j&quot;)
      python3 iprscan5.py \
          --goterms \
          --pathways \
          --email=&quot;blabla@protonmail.com&quot; \
          --outfile=out/&quot;<span class="math-container">$filename" \
      --outformat=tsv \
      --quiet \
      "$</span>j&quot; &amp; (( i++%waitevery==0 )) &amp;&amp; wait
  done
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-08-17T20:02:37.503Z
Answers.Text: I asked them how they're running 'submit.sh` for the reason you picked precisely - they're probably using `sh` and hitting a POSIX compatibility wall.
Answers.UserDisplayName: Ram RS
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-08-17T11:08:10.003Z
Answers.LastActivityDate: 2022-08-17T11:08:10.003Z
Answers.UserDisplayName: terdon
Answers.Body: <p>I'd guess that the specific error that's coming up is because there are spaces missing from the wait line. This seems to be a problem in the source script, i.e.:</p>
  <pre><code>$j &amp; (( i++%waitevery==0 )) &amp;&amp; wait
  </code></pre>
  <p>should probably be:</p>
  <pre><code>$j &amp; (( i++ % waitevery == 0 )) &amp;&amp; wait
  </code></pre>
  <p>but for me (following @terdon's reproduction advice via <code>dash -c &quot;i=9; ((++i%5==0)) &amp;&amp; echo 'worked!'&quot;</code>) that only changes the thing that is not found from <code>i++%waitevery==0</code> to <code>i++</code>, suggesting that the post-increment operator is not supported in the shell. Post-increment <em>is</em> <a href="https://linuxhint.com/bash_operator_examples/#o13" rel="nofollow noreferrer">supported in bash</a>, suggesting that this script is being run (as @RamRS commented) in an unexpected way.</p>
  <p>Try running the script directly:</p>
  <pre><code>./interpro_batch_submit.sh
  </code></pre>
  <p>Or explicitly using bash:</p>
  <pre><code>bash ./interpro_batch_submit.sh
  </code></pre>
  <p>... but see @terdon's answer for a better approach to doing this.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-08-17T11:50:59.717Z
Answers.LastActivityDate: 2022-08-17T21:55:08.610Z
Answers.LastEditDate: 2022-08-17T21:55:08.610Z
Answers.UserDisplayName: gringer
Body: <p>hey i have the problem with bash script. It should run interproscan REST
  Im green in bash scripting. I found this script on the <a href="https://gist.github.com/terrymun/889f28314643d429496881adbdd40039" rel="nofollow noreferrer">web</a>:</p>
  <pre><code>#!/bin/bash
  i=1
  waitevery=30
  
  mkdir -p out
  
  for j in <span class="math-container">$(find `pwd` -type f -name "*.fa")
  do
      echo "Iteration: $</span>i; File: <span class="math-container">$j"
      filename=$</span>(basename &quot;<span class="math-container">$j")
      python3 iprscan5.py \
          --goterms \
          --pathways \
          --email=blabla@protonmail.com \
          --outfile=out/$</span>{filename} \
          --outformat=tsv \
          --quiet \
          $j &amp; (( i++%waitevery==0 )) &amp;&amp; wait
  done
  </code></pre>
  <p>out:</p>
  <pre><code>Iteration: 1; File: /home/marcin/interproscan-5.57-90.0-64-bit/interproscan-5.57-90.0/test/xtr.fa
  interpro_batch_submit.sh: 18: i++%waitevery==0: not found
  Iteration: 1; File: /home/marcin/interproscan-5.57-90.0-64-bit/interproscan-5.57-90.0/test/xts.fa
  interpro_batch_submit.sh: 18: i++%waitevery==0: not found
  Iteration: 1; File: /home/marcin/interproscan-5.57-90.0-64-bit/interproscan-5.57-90.0/test/xtt.fa
  interpro_batch_submit.sh: 18: i++%waitevery==0: not found
  Iteration: 1; File: /home/marcin/interproscan-5.57-90.0-64-bit/interproscan-5.57-90.0/test/xtu.fa
  interpro_batch_submit.sh: 18: i++%waitevery==0: not found
  Iteration: 1; File: /home/marcin/interproscan-5.57-90.0-64-bit/interproscan-5.57-90.0/test/xtv.fa
  interpro_batch_submit.sh: 18: i++%waitevery==0: not found
  Iteration: 1; File: /home/marcin/interproscan-5.57-90.0-64-bit/interproscan-5.57-90.0/test/xtw.fa
  </code></pre>
  <p>Edit:</p>
  <p>I think I forgot to add <code>--sequence</code>, but something was still wrong after it.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-08-16T17:19:43.890Z
Comments.Text: Also, I'm assuming you're saving the script as `submit.sh` and then running it. How are you running it exactly?
Comments.UserDisplayName: Ram RS
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-08-17T11:15:17.800Z
Comments.Text: @RamRS `wait` will wait until all background jobs have finished. `(( i++%waitevery==0 ))` will be true whenever the current value of `i` modulo `waitevery` is 0, so this is a really neat trick to make the script wait for all background processes to execute every N processes. Quite clever, that.
Comments.UserDisplayName: terdon
Tags: <bash>
Title: bash script for interproscan REST
ThreadText: # bash script for interproscan REST
  
  (Asked by: username_0 on 2022-08-16)
  
  hey i have the problem with bash script. It should run interproscan REST
  Im green in bash scripting. I found this script on the [web](https://gist.github.com/terrymun/889f28314643d429496881adbdd40039):
  
  ```
  #!/bin/bash
  i=1
  waitevery=30
  
  mkdir -p out
  
  for j in $(find `pwd` -type f -name "*.fa")
  do
      echo "Iteration: $i; File: $j"
      filename=$(basename "$j")
      python3 iprscan5.py \
          --goterms \
          --pathways \
          --email=blabla@protonmail.com \
          --outfile=out/${filename} \
          --outformat=tsv \
          --quiet \
          $j & (( i++%waitevery==0 )) && wait
  done
  
  ```
  
  out:
  
  ```
  Iteration: 1; File: /home/marcin/interproscan-5.57-90.0-64-bit/interproscan-5.57-90.0/test/xtr.fa
  interpro_batch_submit.sh: 18: i++%waitevery==0: not found
  Iteration: 1; File: /home/marcin/interproscan-5.57-90.0-64-bit/interproscan-5.57-90.0/test/xts.fa
  interpro_batch_submit.sh: 18: i++%waitevery==0: not found
  Iteration: 1; File: /home/marcin/interproscan-5.57-90.0-64-bit/interproscan-5.57-90.0/test/xtt.fa
  interpro_batch_submit.sh: 18: i++%waitevery==0: not found
  Iteration: 1; File: /home/marcin/interproscan-5.57-90.0-64-bit/interproscan-5.57-90.0/test/xtu.fa
  interpro_batch_submit.sh: 18: i++%waitevery==0: not found
  Iteration: 1; File: /home/marcin/interproscan-5.57-90.0-64-bit/interproscan-5.57-90.0/test/xtv.fa
  interpro_batch_submit.sh: 18: i++%waitevery==0: not found
  Iteration: 1; File: /home/marcin/interproscan-5.57-90.0-64-bit/interproscan-5.57-90.0/test/xtw.fa
  
  ```
  
  Edit:
  
  I think I forgot to add `--sequence`, but something was still wrong after it.
  
  **Tags:** bash
  
  > **Comments:**
  > **username_1**: Also, I'm assuming you're saving the script as `submit.sh` and then running it. How are you running it exactly?  
  > **username_2**: @username_1 `wait` will wait until all background jobs have finished. `(( i++%waitevery==0 ))` will be true whenever the current value of `i` modulo `waitevery` is 0, so this is a really neat trick to make the script wait for all background processes to execute every N processes. Quite clever, that.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2022-08-17. Score: 1 | Accepted answer
  
  First of all *never* do `for foo in $(find ...)`. That is very bad practice as it can't handle even slightly odd file names like those with a space. See [bash pitfall #1](https://mywiki.wooledge.org/BashPitfalls#for_f_in_.24.28ls_.2A.mp3.29).
  
  Next, you don't need `pwd`, just run `find -type f -name "*.fa"` which will search in the current directory by default and unlike `find `pwd`...` , won't break if your path contains whitespace.
  
  That's just general best practice, the real issue here is that you are probably running the script like this:
  
  ```
  sh /path/to/script.sh
  
  ```
  
  Right? And you are using a Debian-based system like Ubuntu, right? On such systems, `/bin/sh` is actually not bash but a very basic POSIX compatible shell called `dash` and `dash` doesn't understand the `(( i++%waitevery==0 ))` syntax.
  
  Since you have a shebang line (`#!/bin/bash`), you can simply make your script executable (`chmod a+x /path/to/script.sh`) and then run the script directly like this:
  
  ```
  /path/to/script.sh
  
  ```
  
  Or, alternatively, run it explicitly with `bash` instead of `sh`:
  
  ```
  bash /path/to/script.sh
  
  ```
  
  Either should work and if your file names and paths are sane, the issues I mentioned above won't be relevant. But to keep things nice and clean, I would also change the script to (note that this assumes you are using Linux and GNU `find`):
  
  ```sh
  #!/bin/bash
  i=1
  waitevery=30
  
  mkdir -p out
  
  find -type f -name "*.fa" -print0 |
    while IFS= read -r -d '' j; do 
      echo "Iteration: $i; File: $j"
      filename=$(basename "$j")
      python3 iprscan5.py \
          --goterms \
          --pathways \
          --email="blabla@protonmail.com" \
          --outfile=out/"$filename" \
      --outformat=tsv \
      --quiet \
      "$j" & (( i++%waitevery==0 )) && wait
  done
  
  ```
  
  > **Comments:**
  > **username_1**: I asked them how they're running 'submit.sh` for the reason you picked precisely - they're probably using `sh` and hitting a POSIX compatibility wall.  
  
  ---
  
  ### Answer by username_3 on 2022-08-17. Score: 1
  
  I'd guess that the specific error that's coming up is because there are spaces missing from the wait line. This seems to be a problem in the source script, i.e.:
  
  ```
  $j & (( i++%waitevery==0 )) && wait
  
  ```
  
  should probably be:
  
  ```
  $j & (( i++ % waitevery == 0 )) && wait
  
  ```
  
  but for me (following @username_2's reproduction advice via `dash -c "i=9; ((++i%5==0)) && echo 'worked!'"`) that only changes the thing that is not found from `i++%waitevery==0` to `i++`, suggesting that the post-increment operator is not supported in the shell. Post-increment *is* [supported in bash](https://linuxhint.com/bash_operator_examples/#o13), suggesting that this script is being run (as @username_1 commented) in an unexpected way.
  
  Try running the script directly:
  
  ```
  ./interpro_batch_submit.sh
  
  ```
  
  Or explicitly using bash:
  
  ```
  bash ./interpro_batch_submit.sh
  
  ```
  
  ... but see @username_2's answer for a better approach to doing this.
  
  
  ---
  
  