Answers.Body: <p>I understand (see <strong>Slurm array job solution</strong> below), what I can definitely help with is the automated generation of the <code>seqc.yaml</code> file. This avoids the repetition (and risk of error) in copy and pasting. The following one-liner exactly reproduces the <code>seqc.yaml</code> file and needs the <code>chunk.yaml</code> below.</p>
  <pre><code>for ((x=10;x&lt;=13;x++)); do while IFS='' read -r a; do echo &quot;<span class="math-container">${a//SM-745-10/SM-745-$</span>{x}}&quot;; done &lt; chunk.yaml &gt;&gt; seqc.yaml; done
  </code></pre>
  <p>This could be generalised to 3 command lines in bash,</p>
  <pre><code>y='SM-745-10'
  z='M-745'
  for ((x=10;x&lt;=13;x++)); \
  do while IFS='' read -r a; do echo &quot;<span class="math-container">${a//${y}/${z}$</span>{x}}&quot;;\
  done &lt; chunk.yaml &gt;&gt; seqc.yaml; done
  </code></pre>
  <p>Here everything from <code>for</code> is a bash one-liner. Obviously <code>((x=10,x&lt;=13;x++))</code> would be <code>((x=1,x&lt;=20;x++))</code> for 20 fastq forward and reverse files. This will ensure the separate files are correctly named in the yaml file, in this example, viz.</p>
  <pre><code># first pair
  SM-745-10_R1_001.fastq.gz
  SM-745-10_R2_001.fastq.gz
  # second pair
  SM-745-11_R1_001.fastq.gz
  SM-745-11_R2_001.fastq.gz
  # third pair
  SM-745-12_R1_001.fastq.gz
  SM-745-12_R2_001.fastq.gz
  </code></pre>
  <p>The <code>chunk.yaml</code> is,</p>
  <pre><code>details:
  - algorithm:
      aligner: star
      disambiguate: mm10
      expression_caller:
      - salmon
      - kallisto
      quality_format: standard
      strandedness: auto
      transcriptome_fasta: /swiftcache/ngs/oncology/analysis/todd_creasy/working/reference/GRCh38.primary_assembly.genome_GPC3CART.fa
      transcriptome_gtf: /swiftcache/ngs/oncology/analysis/todd_creasy/working/reference/gencode.v29.annotation_GPC3CART.gtf
    analysis: RNA-seq
    description: 10_LI6612_Baseline_8743
    files:
    - /swiftcache/ngs/oncology/analysis/todd_creasy/working/fastq/SM-745/SM-745-10_R1_001.fastq.gz
    - /swiftcache/ngs/oncology/analysis/todd_creasy/working/fastq/SM-745/SM-745-10_R2_001.fastq.gz
    genome_build: hg38
    metadata:
      panel: Baseline
  -
  </code></pre>
  <hr />
  <h3>bcbio</h3>
  <p>To summarise the <a href="https://github.com/bcbio/bcbio-nextgen/issues/3719" rel="nofollow noreferrer">response from the Git support team</a> is there's conflict occurring  ...</p>
  <ol>
  <li>The issue is here for the slurm script ...</li>
  </ol>
  <pre><code>#SBATCH -p core -n 32
  </code></pre>
  <p>I assume this is <code>-N 1</code> ,i.e. &gt;= 32 core node, if it was 16 cores per node then <code>-N 2</code>.</p>
  <p>However, <strong>bcbio</strong> is taking care of this and the restriction <code>32</code> is conflicting with <code>72</code> from <code>bcbio</code>, i.e. it creates its own Slurm controller script.</p>
  <ol start="2">
  <li>in <strong>bcbio</strong> yaml</li>
  </ol>
  <pre><code>    default:
      memory: 30G
      cores: 72
  </code></pre>
  <p>2a. Firstly <code>72</code> is specified in Slurm's CLI here ... <code>-n 72</code></p>
  <p>2b. This isn't needed because,</p>
  <pre><code>#SBATCH --mem-per-cpu=40G 
  </code></pre>
  <p><code>--mem-per-cpu</code>  This is a very recent feature BTW.</p>
  <hr />
  <h3>Slurm array job solution</h3>
  <p>What I considered was using the <code>chunk.yaml</code> above with an array job - this solution isn't needed because <strong>bcbio</strong> is dynamically reading and parallelising <code>seqc.yaml</code> (maybe deploying a wrapper for <code>sinfo</code>, <code>squeue</code>). In the solution below what I'm doing is placing each fastq pair directly into the Slurm queue. This would then parallelise everything directly. Again whilst it would work, it's unnecessary.</p>
  <p>Thus for all 20 fastq files assuming <code>SM-745-1_R1_001.fastq.gz</code> ... <code>SM-745-20_R1_001.fastq.gz</code>.</p>
  <pre><code>#!/bin/bash
  
  #SBATCH --job-name=bcbiopipeline_GPC3_PDx
  #SBATCH -t 1-00:00:00 # 1- is one day. 00:00:00 is a wall clock. Does &quot;1-00:00&quot; work?
  #SBATCH -c 1
  #SBATCH --array=1-20 # this is now in $SLURM_ARRAY_TASK_ID
  #SBATCH --mem-per-cpu=40G
  #SBATCH --error=job.%J.err
  #SBATCH --output=job.%J.out
  #SBATCH --export=ALL
  
  echo &quot;Load bcbio...&quot;
  module use /projects/ngs_prodifx/local/software/modules/  
  module load bcbio-nextgen/1.2.9 
  
  # SM-745-10 and SM-745 could be assigned as variables y and z as above
  
  while IFS='' read -r a; \
  do echo &quot;<span class="math-container">${a//SM-745-10/SM-745-$</span>SLURM_ARRAY_TASK_ID&quot;; \
  done &lt; chunk.yaml &gt; seqc.yaml; done
  bcbio_nextgen.py seqc.yaml -n 72 -t ipython -s slurm -q core -r t=0-72:00 -r conmem=20 --timeout 3000 --tag &quot;rna&quot;
  
  </code></pre>
  <p><strong>Note</strong> the <code>-n 72</code> parameter, i.e. number of cores deployed, will likely be reduced under this solution.</p>
  <p><strong>Alternative</strong> The alternative is to issue a slight modification of the bash one-liner described above to produce 3 (or 20) separate yaml files via bash <code>while</code> referring the different fastq paired files and use array job to cycle through them and delete each in turn when done.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-27T16:05:14.423Z
Answers.LastActivityDate: 2023-10-30T17:43:36.270Z
Answers.LastEditDate: 2023-10-30T17:43:36.270Z
Answers.UserDisplayName: M__
Body: <p>Sorry for the cross post but I'm not getting a response on bcbio github:</p>
  <p>I'm trying to run bcbio on 20 samples and it's only running one at a time as it steps through the yaml seemingly. I'm using slurm. I was wondering if someone can look at this bcbio snippet and if they see anything wrong?</p>
  <p>Slurm script</p>
  <pre><code>#!/bin/bash
  
  #SBATCH --job-name=bcbiopipeline_GPC3_PDx
  #SBATCH -t 1-00:00
  #SBATCH -c 1
  #SBATCH -p core -n 32
  #SBATCH --mem-per-cpu=40G
  #SBATCH --error=job.%J.err
  #SBATCH --output=job.%J.out
  #SBATCH --export=ALL
  
  echo &quot;Load bcbio...&quot;
  module use /projects/ngs_prodifx/local/software/modules/
  module load bcbio-nextgen/1.2.9
  
  bcbio_nextgen.py seqc.yaml -n 72 -t ipython -s slurm -q core -r t=0-72:00 -r conmem=20 --timeout 3000 --tag &quot;rna&quot;
  
  </code></pre>
  <p>yaml snippet:</p>
  <pre><code>resources:
    default:
      memory: 30G
      cores: 72
  details:
  - algorithm:
      aligner: star
      disambiguate: mm10
      expression_caller:
      - salmon
      - kallisto
      quality_format: standard
      strandedness: auto
      transcriptome_fasta: /swiftcache/ngs/oncology/analysis/todd_creasy/working/reference/GRCh38.primary_assembly.genome_GPC3CART.fa
      transcriptome_gtf: /swiftcache/ngs/oncology/analysis/todd_creasy/working/reference/gencode.v29.annotation_GPC3CART.gtf
    analysis: RNA-seq
    description: 10_LI6612_Baseline_8743
    files:
    - /swiftcache/ngs/oncology/analysis/todd_creasy/working/fastq/SM-745/SM-745-10_R1_001.fastq.gz
    - /swiftcache/ngs/oncology/analysis/todd_creasy/working/fastq/SM-745/SM-745-10_R2_001.fastq.gz
    genome_build: hg38
    metadata:
      panel: Baseline
  - algorithm:
      aligner: star
      disambiguate: mm10
      expression_caller:
      - salmon
      - kallisto
      quality_format: standard
      strandedness: auto
      transcriptome_fasta: /swiftcache/ngs/oncology/analysis/todd_creasy/working/reference/GRCh38.primary_assembly.genome_GPC3CART.fa
      transcriptome_gtf: /swiftcache/ngs/oncology/analysis/todd_creasy/working/reference/gencode.v29.annotation_GPC3CART.gtf
    analysis: RNA-seq
    description: 11_LI6612_Baseline_8765
    files:
    - /swiftcache/ngs/oncology/analysis/todd_creasy/working/fastq/SM-745/SM-745-11_R1_001.fastq.gz
    - /swiftcache/ngs/oncology/analysis/todd_creasy/working/fastq/SM-745/SM-745-11_R2_001.fastq.gz
    genome_build: hg38
    metadata:
      panel: Baseline
  - algorithm:
      aligner: star
      disambiguate: mm10
      expression_caller:
      - salmon
      - kallisto
      quality_format: standard
      strandedness: auto
      transcriptome_fasta: /swiftcache/ngs/oncology/analysis/todd_creasy/working/reference/GRCh38.primary_assembly.genome_GPC3CART.fa
      transcriptome_gtf: /swiftcache/ngs/oncology/analysis/todd_creasy/working/reference/gencode.v29.annotation_GPC3CART.gtf
    analysis: RNA-seq
    description: 12_LI6612_Baseline_8770
    files:
    - /swiftcache/ngs/oncology/analysis/todd_creasy/working/fastq/SM-745/SM-745-12_R1_001.fastq.gz
    - /swiftcache/ngs/oncology/analysis/todd_creasy/working/fastq/SM-745/SM-745-12_R2_001.fastq.gz
    genome_build: hg38
    metadata:
      panel: Baseline
  fc_name: seqc
  upload:
    dir: ../output/final
  </code></pre>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-10-26T15:22:54.083Z
Comments.Text: I recommend this is about Slurm rather than bcbio. I would personally do this as a Slurm job array (just my opinion). This is better to resolve via [chat](https://chat.stackexchange.com/rooms/58858/bioinformatics) in my personal opinion. The CLI in the slurm script is key. I can see the Lmod via **use** and **load** probably via EasyBuild.  What I need to do is read about **bcbio** and see how thats put together.
Comments.UserDisplayName: M__
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-10-26T15:41:43.743Z
Comments.Text: You can see the bcbio docs for slurm here: https://bcbio-nextgen.readthedocs.io/en/latest/contents/parallel.html. The setup seems right according to the doc which is why I'm thinking there's an issue with the yaml
Comments.UserDisplayName: t_creasman
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-10-26T16:39:52.247Z
Comments.Text: Okay, I've got it. One solution is an array job.  This will make it directly run in parallel rather than sequentially. *Background* this is EasyBuild on OpenStack (Swift) ...The GitHub request is here: https://github.com/bcbio/bcbio-nextgen/issues/3719
Comments.UserDisplayName: M__
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-10-26T20:03:15.973Z
Comments.Text: @M__ - I reached out to a bcbio expert and it was just a bad sbatch script. You can see his response here: https://github.com/bcbio/bcbio-nextgen/issues/3719
Comments.UserDisplayName: t_creasman
Tags: <python><slurm><yaml>
Title: bcbio run not running samples in parallel
ThreadText: # bcbio run not running samples in parallel
  
  (Asked by: username_0 on 2023-10-26)
  
  Sorry for the cross post but I'm not getting a response on bcbio github:
  
  I'm trying to run bcbio on 20 samples and it's only running one at a time as it steps through the yaml seemingly. I'm using slurm. I was wondering if someone can look at this bcbio snippet and if they see anything wrong?
  
  Slurm script
  
  ```
  #!/bin/bash
  
  #SBATCH --job-name=bcbiopipeline_GPC3_PDx
  #SBATCH -t 1-00:00
  #SBATCH -c 1
  #SBATCH -p core -n 32
  #SBATCH --mem-per-cpu=40G
  #SBATCH --error=job.%J.err
  #SBATCH --output=job.%J.out
  #SBATCH --export=ALL
  
  echo "Load bcbio..."
  module use /projects/ngs_prodifx/local/software/modules/
  module load bcbio-nextgen/1.2.9
  
  bcbio_nextgen.py seqc.yaml -n 72 -t ipython -s slurm -q core -r t=0-72:00 -r conmem=20 --timeout 3000 --tag "rna"
  
  ```
  
  yaml snippet:
  
  ```
  resources:
    default:
      memory: 30G
      cores: 72
  details:
  - algorithm:
      aligner: star
      disambiguate: mm10
      expression_caller:
      - salmon
      - kallisto
      quality_format: standard
      strandedness: auto
      transcriptome_fasta: /swiftcache/ngs/oncology/analysis/todd_creasy/working/reference/GRCh38.primary_assembly.genome_GPC3CART.fa
      transcriptome_gtf: /swiftcache/ngs/oncology/analysis/todd_creasy/working/reference/gencode.v29.annotation_GPC3CART.gtf
    analysis: RNA-seq
    description: 10_LI6612_Baseline_8743
    files:
    - /swiftcache/ngs/oncology/analysis/todd_creasy/working/fastq/SM-745/SM-745-10_R1_001.fastq.gz
    - /swiftcache/ngs/oncology/analysis/todd_creasy/working/fastq/SM-745/SM-745-10_R2_001.fastq.gz
    genome_build: hg38
    metadata:
      panel: Baseline
  - algorithm:
      aligner: star
      disambiguate: mm10
      expression_caller:
      - salmon
      - kallisto
      quality_format: standard
      strandedness: auto
      transcriptome_fasta: /swiftcache/ngs/oncology/analysis/todd_creasy/working/reference/GRCh38.primary_assembly.genome_GPC3CART.fa
      transcriptome_gtf: /swiftcache/ngs/oncology/analysis/todd_creasy/working/reference/gencode.v29.annotation_GPC3CART.gtf
    analysis: RNA-seq
    description: 11_LI6612_Baseline_8765
    files:
    - /swiftcache/ngs/oncology/analysis/todd_creasy/working/fastq/SM-745/SM-745-11_R1_001.fastq.gz
    - /swiftcache/ngs/oncology/analysis/todd_creasy/working/fastq/SM-745/SM-745-11_R2_001.fastq.gz
    genome_build: hg38
    metadata:
      panel: Baseline
  - algorithm:
      aligner: star
      disambiguate: mm10
      expression_caller:
      - salmon
      - kallisto
      quality_format: standard
      strandedness: auto
      transcriptome_fasta: /swiftcache/ngs/oncology/analysis/todd_creasy/working/reference/GRCh38.primary_assembly.genome_GPC3CART.fa
      transcriptome_gtf: /swiftcache/ngs/oncology/analysis/todd_creasy/working/reference/gencode.v29.annotation_GPC3CART.gtf
    analysis: RNA-seq
    description: 12_LI6612_Baseline_8770
    files:
    - /swiftcache/ngs/oncology/analysis/todd_creasy/working/fastq/SM-745/SM-745-12_R1_001.fastq.gz
    - /swiftcache/ngs/oncology/analysis/todd_creasy/working/fastq/SM-745/SM-745-12_R2_001.fastq.gz
    genome_build: hg38
    metadata:
      panel: Baseline
  fc_name: seqc
  upload:
    dir: ../output/final
  
  ```
  
  **Tags:** python, slurm, yaml
  
  > **Comments:**
  > **username_1**: I recommend this is about Slurm rather than bcbio. I would personally do this as a Slurm job array (just my opinion). This is better to resolve via [chat](https://chat.stackexchange.com/rooms/58858/bioinformatics) in my personal opinion. The CLI in the slurm script is key. I can see the Lmod via **use** and **load** probably via EasyBuild.  What I need to do is read about **bcbio** and see how thats put together.  
  > **username_0**: You can see the bcbio docs for slurm here: https://bcbio-nextgen.readthedocs.io/en/latest/contents/parallel.html. The setup seems right according to the doc which is why I'm thinking there's an issue with the yaml  
  > **username_1**: Okay, I've got it. One solution is an array job.  This will make it directly run in parallel rather than sequentially. *Background* this is EasyBuild on OpenStack (Swift) ...The GitHub request is here: https://github.com/bcbio/bcbio-nextgen/issues/3719  
  > **username_0**: @username_1 - I reached out to a bcbio expert and it was just a bad sbatch script. You can see his response here: https://github.com/bcbio/bcbio-nextgen/issues/3719  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-10-27. Score: 0
  
  I understand (see **Slurm array job solution** below), what I can definitely help with is the automated generation of the `seqc.yaml` file. This avoids the repetition (and risk of error) in copy and pasting. The following one-liner exactly reproduces the `seqc.yaml` file and needs the `chunk.yaml` below.
  
  ```
  for ((x=10;x<=13;x++)); do while IFS='' read -r a; do echo "${a//SM-745-10/SM-745-${x}}"; done < chunk.yaml >> seqc.yaml; done
  
  ```
  
  This could be generalised to 3 command lines in bash,
  
  ```
  y='SM-745-10'
  z='M-745'
  for ((x=10;x<=13;x++)); \
  do while IFS='' read -r a; do echo "${a//${y}/${z}${x}}";\
  done < chunk.yaml >> seqc.yaml; done
  
  ```
  
  Here everything from `for` is a bash one-liner. Obviously `((x=10,x<=13;x++))` would be `((x=1,x<=20;x++))` for 20 fastq forward and reverse files. This will ensure the separate files are correctly named in the yaml file, in this example, viz.
  
  ```
  # first pair
  SM-745-10_R1_001.fastq.gz
  SM-745-10_R2_001.fastq.gz
  # second pair
  SM-745-11_R1_001.fastq.gz
  SM-745-11_R2_001.fastq.gz
  # third pair
  SM-745-12_R1_001.fastq.gz
  SM-745-12_R2_001.fastq.gz
  
  ```
  
  The `chunk.yaml` is,
  
  ```
  details:
  - algorithm:
      aligner: star
      disambiguate: mm10
      expression_caller:
      - salmon
      - kallisto
      quality_format: standard
      strandedness: auto
      transcriptome_fasta: /swiftcache/ngs/oncology/analysis/todd_creasy/working/reference/GRCh38.primary_assembly.genome_GPC3CART.fa
      transcriptome_gtf: /swiftcache/ngs/oncology/analysis/todd_creasy/working/reference/gencode.v29.annotation_GPC3CART.gtf
    analysis: RNA-seq
    description: 10_LI6612_Baseline_8743
    files:
    - /swiftcache/ngs/oncology/analysis/todd_creasy/working/fastq/SM-745/SM-745-10_R1_001.fastq.gz
    - /swiftcache/ngs/oncology/analysis/todd_creasy/working/fastq/SM-745/SM-745-10_R2_001.fastq.gz
    genome_build: hg38
    metadata:
      panel: Baseline
  -
  
  ```
  
  ---
  
  ### bcbio
  
  To summarise the [response from the Git support team](https://github.com/bcbio/bcbio-nextgen/issues/3719) is there's conflict occurring ...
  
  1. The issue is here for the slurm script ...
  
  ```
  #SBATCH -p core -n 32
  
  ```
  
  I assume this is `-N 1` ,i.e. >= 32 core node, if it was 16 cores per node then `-N 2`.
  
  However, **bcbio** is taking care of this and the restriction `32` is conflicting with `72` from `bcbio`, i.e. it creates its own Slurm controller script.
  
  2. in **bcbio** yaml
  
  ```
      default:
      memory: 30G
      cores: 72
  
  ```
  
  2a. Firstly `72` is specified in Slurm's CLI here ... `-n 72`
  
  2b. This isn't needed because,
  
  ```
  #SBATCH --mem-per-cpu=40G 
  
  ```
  
  `--mem-per-cpu` This is a very recent feature BTW.
  
  ---
  
  ### Slurm array job solution
  
  What I considered was using the `chunk.yaml` above with an array job - this solution isn't needed because **bcbio** is dynamically reading and parallelising `seqc.yaml` (maybe deploying a wrapper for `sinfo`, `squeue`). In the solution below what I'm doing is placing each fastq pair directly into the Slurm queue. This would then parallelise everything directly. Again whilst it would work, it's unnecessary.
  
  Thus for all 20 fastq files assuming `SM-745-1_R1_001.fastq.gz` ... `SM-745-20_R1_001.fastq.gz`.
  
  ```
  #!/bin/bash
  
  #SBATCH --job-name=bcbiopipeline_GPC3_PDx
  #SBATCH -t 1-00:00:00 # 1- is one day. 00:00:00 is a wall clock. Does "1-00:00" work?
  #SBATCH -c 1
  #SBATCH --array=1-20 # this is now in $SLURM_ARRAY_TASK_ID
  #SBATCH --mem-per-cpu=40G
  #SBATCH --error=job.%J.err
  #SBATCH --output=job.%J.out
  #SBATCH --export=ALL
  
  echo "Load bcbio..."
  module use /projects/ngs_prodifx/local/software/modules/  
  module load bcbio-nextgen/1.2.9 
  
  # SM-745-10 and SM-745 could be assigned as variables y and z as above
  
  while IFS='' read -r a; \
  do echo "${a//SM-745-10/SM-745-$SLURM_ARRAY_TASK_ID"; \
  done < chunk.yaml > seqc.yaml; done
  bcbio_nextgen.py seqc.yaml -n 72 -t ipython -s slurm -q core -r t=0-72:00 -r conmem=20 --timeout 3000 --tag "rna"
  
  ```
  
  **Note** the `-n 72` parameter, i.e. number of cores deployed, will likely be reduced under this solution.
  
  **Alternative** The alternative is to issue a slight modification of the bash one-liner described above to produce 3 (or 20) separate yaml files via bash `while` referring the different fastq paired files and use array job to cycle through them and delete each in turn when done.
  
  
  ---
  
  