Answers.Body: <p>What's best is quite subjective. My preference would be use <a href="https://www.nextflow.io/" rel="noreferrer">Nextflow</a> to abstract away the underlying job submission system and then use the <a href="https://www.nextflow.io/docs/latest/executor.html#sge" rel="noreferrer">sge</a> executor to submit the jobs to the cluster. The only issue I see, is that it looks like you're feeding in a list of sample IDs, which your script presumably uses to find some VCF files containing the variants. If the script could be refactored to accept file inputs, we can have Nextflow localize them into its working directory. This would ensure your workflow (however trivial) is portable and could be run on AWS Batch, for example. The following example uses the <a href="https://www.nextflow.io/docs/latest/operator.html#combine" rel="noreferrer">combine</a> operator to create the cartesian product and the <a href="https://www.nextflow.io/docs/latest/operator.html#collectfile" rel="noreferrer">collectFile</a> operator to gather the results. With the following in a file called <code>main.nf</code>:</p>
  <pre><code>params.vcf_files = './path/to/*.vcf'
  
  
  process compare_variants {
  
      input:
      tuple(
          path(sample_A_vcf, stageAs: 'A/*'),
          path(sample_B_vcf, stageAs: 'B/*'),
      ) 
  
      output:
      stdout
  
      &quot;&quot;&quot;
      your_script.py \\
          --sample-vcf-A &quot;<span class="math-container">${sample_A_vcf}" \\
          --sample-vcf-B "$</span>{sample_B_vcf}&quot; 
      &quot;&quot;&quot;
  }
  
  workflow {
       
       samples = Channel.fromPath( params.vcf_files )
       product = samples.combine(samples)
  
       compare_variants( product )
           .collectFile( name: 'results.txt' )
           .view { &quot;Saved results to: $it&quot; }
  }
  </code></pre>
  <p>And the following in a file called <code>nextflow.config</code>:</p>
  <pre><code>process {
  
    executor = 'sge'
  }
  </code></pre>
  <p>And your Python script in a folder called 'bin' in the root directory of your project repository. Nextflow automatically adds this folder to the $PATH in the execution environment. So make it executable, for example using <code>chmod +x your_script.py</code>. Then, with Nextflow installed give it a test:</p>
  <pre><code>$ nextflow run main.nf
  N E X T F L O W  ~  version 22.10.7
  Launching `main.nf` [chaotic_watson] DSL2 - revision: ac224d863b
  executor &gt;  local (484)
  [bc/a39b27] process &gt; compare_variants (482) [100%] 484 of 484 ✔
  Saved results to: /path/to/work/tmp/3e/95330f32edec3f109b197bc836b2ca/results.txt
  
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-03T18:10:24.323Z
Answers.Text: Thanks Steve! I'm going to try using nextflow. What is stdout referring to? I am getting an error message in my log that this is an unknown variable.
Answers.UserDisplayName: che625
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-03T18:12:29.617Z
Answers.Text: Also, where would I enter in my array info? e.g. the sample IDs?
Answers.UserDisplayName: che625
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-03T19:59:14.110Z
Answers.Text: Steve, I am getting the error message "No such variable: stdout". Any suggestions?
Answers.UserDisplayName: che625
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-03T23:10:17.567Z
Answers.Text: @che625 The above uses the [`stdout`](https://www.nextflow.io/docs/latest/process.html#output-type-stdout) qualifier, but you could also have your script redirect stdout to a file and then use the [`path`](https://www.nextflow.io/docs/latest/process.html#output-type-path) qualifier to output the file. You would obtain the sample IDs by either parsing the VCF filenames, or by parsing a CSV containing the sample IDs. For the latter, you could use the [`splitCsv`](https://www.nextflow.io/docs/latest/operator.html#splitcsv) operator.
Answers.UserDisplayName: Steve
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-31T15:27:42.463Z
Answers.LastActivityDate: 2023-03-31T15:27:42.463Z
Answers.UserDisplayName: Steve
Answers.Body: <p>Consider using a workflow language such as Snakemake, Nextflow or Cromwell to handle distributing the jobs for you. It will take a little more time investment upfront to set them up, but it will save you many hours and headaches later on.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-01T13:50:13.647Z
Answers.Text: This is a good suggestion, but as it stands, this is a comment not an answer. How would the OP convert their script to one of your suggestions? What would it look like? Please see [Your answer is in another castle: when is an answer not an answer?](https://meta.stackexchange.com/q/225370)
Answers.UserDisplayName: terdon
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-31T10:11:57.803Z
Answers.LastActivityDate: 2023-03-31T10:11:57.803Z
Answers.UserDisplayName: Pallie
Answers.Body: <p>You may use <a href="https://github.com/lh3/asub/blob/master/asub" rel="nofollow noreferrer">asub</a> (array job submitter) for this. Modify your script to <code>myscript.sh</code>:</p>
  <pre class="lang-bash prettyprint-override"><code>array=( A037375001 A037646101 A037710601 A037710701 A037753001 A037844701 A038153001 A038172201 A038184401 A038272502 A038298602 A038533901 A038797201 A038944801 A038973401 A039034701 A039048101 A039089601 A039164101 A039303301 A039323801 A039538301 )
  for i in &quot;${array[@]}&quot;; do
    for j in &quot;${array[@]}&quot;; do
      echo &quot;python3 sample_script.py --run-sample-id-A $i --run-sample-id-B $j &gt; out-$i-$j.csv&quot;
    done
  done
  </code></pre>
  <p>Note that the main difference is this script prints command lines, not executing them. You then run asub with</p>
  <pre class="lang-sh prettyprint-override"><code>bash myscript.sh | asub -q myqueue -j myjobname
  </code></pre>
  <p>asub will submit a job array. If you don't like asub, you may submit with</p>
  <pre class="lang-sh prettyprint-override"><code>bash myscript.sh | xargs -i echo &quot;echo '{}' | qsub -q myqueue&quot; | sh
  </code></pre>
  <p>This will submit separate jobs, not a job array. It will be slower and less convenient but you don't need to install anything or learn anything.</p>
  <p>With either approach, you need to run the following to collate all the results:</p>
  <pre class="lang-sh prettyprint-override"><code>cat out-*-*.csv &gt; our.csv
  </code></pre>
  <hr/>
  <p>On a side note, this approach can be adapted for <a href="https://www.gnu.org/software/parallel/" rel="nofollow noreferrer">GNU parallel</a> to spawn multiple processes on the same machine:</p>
  <pre class="lang-sh prettyprint-override"><code>bash myscript.sh | parallel -j 16
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-03T10:17:22.487Z
Answers.Text: GNU parallel can also send processes to multiple remote machines, it isn't only for parallelizing on a single machine. See https://www.gnu.org/software/parallel/parallel_examples.html#example-using-remote-computers
Answers.UserDisplayName: terdon
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-05T16:57:39.890Z
Answers.Text: I should clarify that I'm looking to parallelize across samples, so split up running a certain number of samples on different nodes and then concatenate the results. I think parallel will just run the same script on X nodes. Can anyone provide guidance on this?
Answers.UserDisplayName: che625
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-05T21:56:24.563Z
Answers.Text: @che625 I know what you want to achieve. In my answer, parallel executes each line, in parallel, on the same node. I wouldn't recommend to use parallel across node.
Answers.UserDisplayName: user172818
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-06T16:31:44.073Z
Answers.Text: So should I divide up my input by positions if I want to use parallel?
Answers.UserDisplayName: che625
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-06T22:01:15.247Z
Answers.Text: @che625 I am not sure what you mean. Anyway, for your original question, just use the "parallel" command line I showed at the end.
Answers.UserDisplayName: user172818
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-31T19:04:48.567Z
Answers.LastActivityDate: 2023-04-01T13:52:24.453Z
Answers.LastEditDate: 2023-04-01T13:52:24.453Z
Answers.UserDisplayName: user172818
Answers.Body: <p><strong>Background</strong> Just for everyone else <code>SGE</code> = Sun Grid Engine and is the <code>qsub</code> system mentioned in the title of the question.</p>
  <h3><code>qsub</code>:</h3>
  <p>If by parallelising you mean multithreading a single job across loads of cores then the first thing is how many cores per node are there?</p>
  <p>Usually it is around 10 (but could be much more). That exact number is super important to know because if you request more cores than exist on a single node the job will never run. At best qsub will refuse the job, at worst it will forever be stuck in the queue. For multi-processing on <code>qsub</code> its ...</p>
  <pre><code>qsub -pe omp 8  myscript.sh
  </code></pre>
  <p>Here its requesting 8 cores and myscript.sh will contain the shell command to run the python script. If there's a node with 10-cores 2 are in use, it will then load the job to give its max. capacity.</p>
  <p>Thus, <code>qsub</code> does not do <strong>course grain-parallelisation</strong>, that is <code>MPI</code>. Thus <code>qsub</code> will only parallelise across the cores in a single <code>node</code>. This is without question a limitation for cluster computing.</p>
  <p><code>qsub</code> is more than just submitting to a queue, you need to monitor the queuing system pre- and post- submission. Firstly, this is for the number of existing jobs across that system. Secondly, this is to see whats happening to your stuff.</p>
  <p><code>qstat</code> is the way to understand whats there:</p>
  <pre><code>qstat -f # whats what on the cluster
  qstat -q
  </code></pre>
  <p>This lists all the queues.</p>
  <pre><code>qstat -u username # looks at what a given user is doing
  qstat long-queue
  </code></pre>
  <p>Targets availability on a specific queue.</p>
  <p>A complete list of <code>qstat</code> is <a href="http://docs.adaptivecomputing.com/torque/4-1-3/Content/topics/commands/qstat.htm" rel="nofollow noreferrer">here</a>. Also <code>qusage -l</code> is useful.</p>
  <hr />
  <h3>&quot;Parallelising&quot; loops</h3>
  <p>If you are submitting loads of jobs which are all working independently in parallel it better to use a different strategy on <code>qsub</code>. It is better to submit each job separately to the queue via a <code>qsub</code> loop. Thus there would be two scripts the first is the script in the question, the second is a submission loop: this would comprise a <code>qsub</code> argument within a loop that would submit each job sequentially. Thus the array might be better in the <code>qsub</code> submission loop.</p>
  <p>Monitoring the queue takes place as per normal via <code>qstat</code>.</p>
  <h4>Rationale</h4>
  <p>The reason for this is the way the queue works <code>qsub</code> prioritises single jobs over multi-threaded parallelisation, thus you get your results much quicker.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-05T16:34:45.773Z
Answers.Text: I should clarify that I'm looking to parallelize across samples, so split up running a certain number of samples on different nodes and then concatenate the results. I think parallel will just run the same script on X nodes. Can anyone provide guidance on this?
Answers.UserDisplayName: che625
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-31T12:52:54.183Z
Answers.LastActivityDate: 2023-03-31T14:20:32.560Z
Answers.LastEditDate: 2023-03-31T14:20:32.560Z
Answers.UserDisplayName: M__
Body: <p>I have a bash script that I would like to parellelize to run on multiple nodes. My goal is to run my python sample_script.py script on pairwise comparisons of samples to see if their variants are a match or not (output is written by echo for the function so we can compare results). I unfortunately can’t post the contents of the Python script, but it is basically comparing germline variants between samples to determine if a sample is a match or not. I’m trying to figure out how to make the script run faster or run multiple pair wise comparisons at once then concatenate results. The samples I'm comparing are listed in the array. Running the script on multiple samples takes too long, so I'd like to qsub the process or run the script on multiple nodes, then concatenate the results. Can someone advise on the best way to tackle this? I don't have much experience with jobs using qsub. Should I consider an SGE array job and if so how would I go about this?</p>
  <p>Example code below using bash within jupyter notebook:</p>
  <pre><code>%%bash
  array=( A037375001 A037646101 A037710601 )
  for i in &quot;<span class="math-container">${array[@]}"
        do 
          for j in "$</span>{array[@]}&quot;
          do
          python3 sample_script.py --run-sample-id-A &quot;<span class="math-container">$i" --run-sample-id-B "$</span>j&quot;
          echo run-sample-id-A &quot;<span class="math-container">$i" run-sample-id-B "$</span>j&quot;
        done
      done &gt; &quot;/outputdir/out.csv&quot;
  </code></pre>
  <p>Thank you in advance!</p>
  
Tags: <python><bash><parallel>
Title: Parallelize or qsub a bash script
ThreadText: # Parallelize or qsub a bash script
  
  (Asked by: username_0 on 2023-03-31)
  
  I have a bash script that I would like to parellelize to run on multiple nodes. My goal is to run my python sample\_script.py script on pairwise comparisons of samples to see if their variants are a match or not (output is written by echo for the function so we can compare results). I unfortunately can’t post the contents of the Python script, but it is basically comparing germline variants between samples to determine if a sample is a match or not. I’m trying to figure out how to make the script run faster or run multiple pair wise comparisons at once then concatenate results. The samples I'm comparing are listed in the array. Running the script on multiple samples takes too long, so I'd like to qsub the process or run the script on multiple nodes, then concatenate the results. Can someone advise on the best way to tackle this? I don't have much experience with jobs using qsub. Should I consider an SGE array job and if so how would I go about this?
  
  Example code below using bash within jupyter notebook:
  
  ```
  %%bash
  array=( A037375001 A037646101 A037710601 )
  for i in "${array[@]}"
        do 
          for j in "${array[@]}"
          do
          python3 sample_script.py --run-sample-id-A "$i" --run-sample-id-B "$j"
          echo run-sample-id-A "$i" run-sample-id-B "$j"
        done
      done > "/outputdir/out.csv"
  
  ```
  
  Thank you in advance!
  
  **Tags:** python, bash, parallel
  
  
  ---
  
  ## 4 answers
  
  ### Answer by username_1 on 2023-03-31. Score: 5
  
  What's best is quite subjective. My preference would be use [Nextflow](https://www.nextflow.io/) to abstract away the underlying job submission system and then use the [sge](https://www.nextflow.io/docs/latest/executor.html#sge) executor to submit the jobs to the cluster. The only issue I see, is that it looks like you're feeding in a list of sample IDs, which your script presumably uses to find some VCF files containing the variants. If the script could be refactored to accept file inputs, we can have Nextflow localize them into its working directory. This would ensure your workflow (however trivial) is portable and could be run on AWS Batch, for example. The following example uses the [combine](https://www.nextflow.io/docs/latest/operator.html#combine) operator to create the cartesian product and the [collectFile](https://www.nextflow.io/docs/latest/operator.html#collectfile) operator to gather the results. With the following in a file called `main.nf`:
  
  ```
  params.vcf_files = './path/to/*.vcf'
  
  process compare_variants {
  
      input:
      tuple(
          path(sample_A_vcf, stageAs: 'A/*'),
          path(sample_B_vcf, stageAs: 'B/*'),
      ) 
  
      output:
      stdout
  
      """
      your_script.py \\
          --sample-vcf-A "${sample_A_vcf}" \\
          --sample-vcf-B "${sample_B_vcf}" 
      """
  }
  
  workflow {
       
       samples = Channel.fromPath( params.vcf_files )
       product = samples.combine(samples)
  
       compare_variants( product )
           .collectFile( name: 'results.txt' )
           .view { "Saved results to: $it" }
  }
  
  ```
  
  And the following in a file called `nextflow.config`:
  
  ```
  process {
  
    executor = 'sge'
  }
  
  ```
  
  And your Python script in a folder called 'bin' in the root directory of your project repository. Nextflow automatically adds this folder to the $PATH in the execution environment. So make it executable, for example using `chmod +x your_script.py`. Then, with Nextflow installed give it a test:
  
  ```
  $ nextflow run main.nf
  N E X T F L O W  ~  version 22.10.7
  Launching `main.nf` [chaotic_watson] DSL2 - revision: ac224d863b
  executor >  local (484)
  [bc/a39b27] process > compare_variants (482) [100%] 484 of 484 ✔
  Saved results to: /path/to/work/tmp/3e/95330f32edec3f109b197bc836b2ca/results.txt
  
  ```
  
  > **Comments:**
  > **username_0**: Thanks username_1! I'm going to try using nextflow. What is stdout referring to? I am getting an error message in my log that this is an unknown variable.  
  > **username_0**: Also, where would I enter in my array info? e.g. the sample IDs?  
  > **username_0**: username_1, I am getting the error message "No such variable: stdout". Any suggestions?  
  > **username_1**: @username_0 The above uses the [`stdout`](https://www.nextflow.io/docs/latest/process.html#output-type-stdout) qualifier, but you could also have your script redirect stdout to a file and then use the [`path`](https://www.nextflow.io/docs/latest/process.html#output-type-path) qualifier to output the file. You would obtain the sample IDs by either parsing the VCF filenames, or by parsing a CSV containing the sample IDs. For the latter, you could use the [`splitCsv`](https://www.nextflow.io/docs/latest/operator.html#splitcsv) operator.  
  
  ---
  
  ### Answer by username_2 on 2023-03-31. Score: 2
  
  Consider using a workflow language such as Snakemake, Nextflow or Cromwell to handle distributing the jobs for you. It will take a little more time investment upfront to set them up, but it will save you many hours and headaches later on.
  
  > **Comments:**
  > **username_3**: This is a good suggestion, but as it stands, this is a comment not an answer. How would the OP convert their script to one of your suggestions? What would it look like? Please see [Your answer is in another castle: when is an answer not an answer?](https://meta.stackexchange.com/q/225370)  
  
  ---
  
  ### Answer by username_4 on 2023-03-31. Score: 2
  
  You may use [asub](https://github.com/lh3/asub/blob/master/asub) (array job submitter) for this. Modify your script to `myscript.sh`:
  
  ```bash
  array=( A037375001 A037646101 A037710601 A037710701 A037753001 A037844701 A038153001 A038172201 A038184401 A038272502 A038298602 A038533901 A038797201 A038944801 A038973401 A039034701 A039048101 A039089601 A039164101 A039303301 A039323801 A039538301 )
  for i in "${array[@]}"; do
    for j in "${array[@]}"; do
      echo "python3 sample_script.py --run-sample-id-A $i --run-sample-id-B $j > out-$i-$j.csv"
    done
  done
  
  ```
  
  Note that the main difference is this script prints command lines, not executing them. You then run asub with
  
  ```sh
  bash myscript.sh | asub -q myqueue -j myjobname
  
  ```
  
  asub will submit a job array. If you don't like asub, you may submit with
  
  ```sh
  bash myscript.sh | xargs -i echo "echo '{}' | qsub -q myqueue" | sh
  
  ```
  
  This will submit separate jobs, not a job array. It will be slower and less convenient but you don't need to install anything or learn anything.
  
  With either approach, you need to run the following to collate all the results:
  
  ```sh
  cat out-*-*.csv > our.csv
  
  ```
  
  ---
  
  On a side note, this approach can be adapted for [GNU parallel](https://www.gnu.org/software/parallel/) to spawn multiple processes on the same machine:
  
  ```sh
  bash myscript.sh | parallel -j 16
  
  ```
  
  > **Comments:**
  > **username_3**: GNU parallel can also send processes to multiple remote machines, it isn't only for parallelizing on a single machine. See https://www.gnu.org/software/parallel/parallel_examples.html#example-using-remote-computers  
  > **username_0**: I should clarify that I'm looking to parallelize across samples, so split up running a certain number of samples on different nodes and then concatenate the results. I think parallel will just run the same script on X nodes. Can anyone provide guidance on this?  
  > **username_4**: @username_0 I know what you want to achieve. In my answer, parallel executes each line, in parallel, on the same node. I wouldn't recommend to use parallel across node.  
  > **username_0**: So should I divide up my input by positions if I want to use parallel?  
  > **username_4**: @username_0 I am not sure what you mean. Anyway, for your original question, just use the "parallel" command line I showed at the end.  
  
  ---
  
  ### Answer by username_5 on 2023-03-31. Score: 0
  
  **Background** Just for everyone else `SGE` = Sun Grid Engine and is the `qsub` system mentioned in the title of the question.
  
  ### `qsub`:
  
  If by parallelising you mean multithreading a single job across loads of cores then the first thing is how many cores per node are there?
  
  Usually it is around 10 (but could be much more). That exact number is super important to know because if you request more cores than exist on a single node the job will never run. At best qsub will refuse the job, at worst it will forever be stuck in the queue. For multi-processing on `qsub` its ...
  
  ```
  qsub -pe omp 8  myscript.sh
  
  ```
  
  Here its requesting 8 cores and myscript.sh will contain the shell command to run the python script. If there's a node with 10-cores 2 are in use, it will then load the job to give its max. capacity.
  
  Thus, `qsub` does not do **course grain-parallelisation**, that is `MPI`. Thus `qsub` will only parallelise across the cores in a single `node`. This is without question a limitation for cluster computing.
  
  `qsub` is more than just submitting to a queue, you need to monitor the queuing system pre- and post- submission. Firstly, this is for the number of existing jobs across that system. Secondly, this is to see whats happening to your stuff.
  
  `qstat` is the way to understand whats there:
  
  ```
  qstat -f # whats what on the cluster
  qstat -q
  
  ```
  
  This lists all the queues.
  
  ```
  qstat -u username # looks at what a given user is doing
  qstat long-queue
  
  ```
  
  Targets availability on a specific queue.
  
  A complete list of `qstat` is [here](http://docs.adaptivecomputing.com/torque/4-1-3/Content/topics/commands/qstat.htm). Also `qusage -l` is useful.
  
  ---
  
  ### "Parallelising" loops
  
  If you are submitting loads of jobs which are all working independently in parallel it better to use a different strategy on `qsub`. It is better to submit each job separately to the queue via a `qsub` loop. Thus there would be two scripts the first is the script in the question, the second is a submission loop: this would comprise a `qsub` argument within a loop that would submit each job sequentially. Thus the array might be better in the `qsub` submission loop.
  
  Monitoring the queue takes place as per normal via `qstat`.
  
  #### Rationale
  
  The reason for this is the way the queue works `qsub` prioritises single jobs over multi-threaded parallelisation, thus you get your results much quicker.
  
  > **Comments:**
  > **username_0**: I should clarify that I'm looking to parallelize across samples, so split up running a certain number of samples on different nodes and then concatenate the results. I think parallel will just run the same script on X nodes. Can anyone provide guidance on this?  
  
  ---
  
  