Answers.Body: <p>I have some time now, so I'm going to take a shot at this, @M__</p>
  <p>You need to run spades per read pair. When you specify <code>*_1.fastq.gz</code> as the <code>-1</code> param, you're giving it <em>every</em> <code>_1</code> file at once, not one <code>_1</code> file per spades command.</p>
  <p>I'd recommend this:</p>
  <pre><code>for smpl in <span class="math-container">$(ls *_1.fastq.gz | xargs -I infile basename infile _1.fastq.gz)
  do
      cat &gt; "./$</span>{smpl}_spades.slurm&quot; &lt;&lt;EOF
  #!/bin/bash  
   
  #SBATCH --job-name=pangenome_spades.${smpl}  
  #SBATCH --account=lilianasalvador
  #SBATCH --partition=standard
  #SBATCH --nodes=1
  #SBATCH --ntasks=1
  #SBATCH --mem-per-cpu=5gb
  #SBATCH --time=02:00:00
  
  spades.py -1 <span class="math-container">${smpl}_1.fastq.gz -2 $</span>{smpl}_2.fastq.gz -o groups/lilianasalvador/noah_100/SPAdes_output_${smpl} -t 16 --careful
  EOF
  done
  </code></pre>
  <p>Run a sanity check on a few of the <code>.slurm</code> files, then</p>
  <pre><code>ls *.slurm | xargs -I job_file sbatch job_file
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-05-30T03:16:08.877Z
Answers.Text: Cool, thanks. I was going to solve this via a formal Slurm array job, but this will work.
Answers.UserDisplayName: M__
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-05-30T13:26:01.230Z
Answers.Text: Please do add an array-based answer, I'm not familiar with arrays but would love to learn
Answers.UserDisplayName: Ram RS
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-06-06T19:00:04.813Z
Answers.Text: Thank you! This worked but we're only getting one contigs.fasta file when we run. Should we be getting a contigs.fasta per sequence pair or one for all the sequences we ran?
Answers.UserDisplayName: Lucy Kelly
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-06-07T20:46:22.140Z
Answers.Text: I am not familiar with spades but going by M__'s answer, you need something at the `-o` param as well. I'll edit my answer but you should test running 2 samples at the same time before running all of the scripts concurrently.
Answers.UserDisplayName: Ram RS
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-05-30T01:05:52.363Z
Answers.LastActivityDate: 2024-06-07T20:47:31.277Z
Answers.LastEditDate: 2024-06-07T20:47:31.277Z
Answers.UserDisplayName: Ram RS
Answers.Body: <p>The <em>job array</em> is a formal approach to solving this via SLURM</p>
  <p>What I am not clear about is the number of files <code>SRR8599992_1.fastq.gz</code> <code>SRR8599992_2.fastq.gaz</code> I originally thought the final number of files was around 100, I've guessed at 120</p>
  <pre><code>#SBATCH --job-name=pangenome_spade
  #SBATCH --account=lilianasalvador
  #SBATCH --ntasks=120
  #SBATCH --cpus-per-task=16 
  #SBATCH --mem-per-cpu=5gb  
  #SBATCH --time=02:00:00 
  #SBATCH --array=599992-5600100
  
  module load python/3.9  
  module load spades/3.15.0-hpc 
  
  spades.py -1 'SRR'<span class="math-container">${SLURM_ARRAY_TASK_ID}'_1.fastq.gz' -2 'SRR'$</span>{<span class="math-container">$SLURM_ARRAY_TASK_ID}'_2.fastq.gz' -o /groups/lilianasalvador/noah_100/'SPAdes_output_'$</span>{SLURM_ARRAY_TASK_ID}'.fastq' -t 16 --careful 
  </code></pre>
  <p>I would test this via an <code>echo</code> test script writing to a specific file to make sure the <code>spades.py</code> command produces the correct command line for each iteration.</p>
  <p>Things to note ...</p>
  <ul>
  <li>Array jobs is specified by the <code>$SLURM_ARRAY_TASK_ID</code> variable and <code>#SBATCH --array=599992-5600100</code></li>
  <li><code>ntasks</code> is the total number of spades assemblies you are performing here I guessed at 120</li>
  <li>You are parallelising via <code>-t 16</code> therefore this must be reflected in your submission job via <code>--cpus-per-task</code></li>
  <li>The <em>ouput</em> file <em>MUST</em> use <code>$SLURM_ARRAY_TASK_ID</code> otherwise you will continually override your out file.</li>
  </ul>
  <hr />
  <p>This is not an optimal SLURM job because ...</p>
  <ul>
  <li><code>-t 16</code> you need to check with your HPC engineer, this might take a long time to get through the queue and an alternative architecture to permit 16 CPUs can get onto the cluster. HTC users will jump ahead of you, so on a hypervisor setup you can specify a separate cluster for your work. This is not well explained but hope it makes sense.</li>
  <li>Shifting to high throughput computing (HTC) could be considered here to get this through a lot quicker, e.g. this is 120 sequential assemblies, 10 sequential assemblies for 12 separate submissions would be a lot quicker (you just alter the &quot;array job&quot; inputs range.</li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-06-01T17:10:17.953Z
Answers.LastActivityDate: 2024-06-02T03:22:50.020Z
Answers.LastEditDate: 2024-06-02T03:22:50.020Z
Answers.UserDisplayName: M__
Body: <p>I am trying to run a batch script to assemble sequences using the spades.py command on HPC. I tested the script with one sequence (1 forward and 1 reverse read) and this worked. I have 100 sequences in total (200 files) and am trying to assemble all of them without having to resubmit the script each time. This is the original script that worked:</p>
  <pre><code>#!/bin/bash  
   
  #SBATCH --job-name=pangenome_spades  
  #SBATCH --account=lilianasalvador  
  #SBATCH --partition=standard  
  #SBATCH --nodes=1  
  #SBATCH --ntasks=1  
  #SBATCH --mem-per-cpu=5gb  
  #SBATCH --time=01:00:00  
   
  spades.py -1 SRR8599992_1.fastq.gz -2 SRR8599992_2.fastq.gz -o /home/u32/lucykelly/spades_test -t 16 --careful 
  </code></pre>
  <p>This is the script that I've been trying to run and the error it's giving me:</p>
  <pre><code>#!/bin/bash  
   
  #SBATCH --job-name=pangenome_spades  
  #SBATCH --account=lilianasalvador  
  #SBATCH --partition=standard  
  #SBATCH --nodes=1  
  #SBATCH --ntasks=1  
  #SBATCH --mem-per-cpu=5gb  
  #SBATCH --time=02:00:00  
   
  module load python/3.9  
  module load spades/3.15.0-hpc 
  
  spades.py -1 *_1.fastq.gz -2 *_2.fastq.gz -o /groups/lilianasalvador/noah_100/SPAdes_output -t 16 --careful 
  
  SPAdes genome assembler v3.15.0-hpc 
   
  Usage: spades.py [options] -o &lt;output_dir&gt; 
  spades.py: error: Please specify option (e.g. -1, -2, -s, etc)) for the following paths: SRR8599995_1.fastq.gz, SRR8599997_1.fastq.gz, SRR8599998_1.fastq.gz, SRR8600006_1.fastq.gz, SRR8600007_1.fastq.gz, SRR8600008_1.fastq.gz, SRR8600014_1.fastq.gz, SRR8600015_1.fastq.gz, SRR8600020_1.fastq.gz, SRR8600022_1.fastq.gz, SRR8600026_1.fastq.gz, SRR8600031_1.fastq.gz, SRR8600044_1.fastq.gz, SRR8600045_1.fastq.gz, SRR8600048_1.fastq.gz, SRR8600049_1.fastq.gz, SRR8600052_1.fastq.gz, SRR8600055_1.fastq.gz, SRR8600058_1.fastq.gz, SRR8600059_1.fastq.gz, SRR8600064_1.fastq.gz, SRR8600070_1.fastq.gz, SRR8600072_1.fastq.gz, SRR8600075_1.fastq.gz, SRR8600079_1.fastq.gz, SRR8600092_1.fastq.gz, SRR8600093_1.fastq.gz, SRR8600098_1.fastq.gz, SRR8600100_1.fastq.gz, SRR8600101_1.fastq.gz, SRR8600102_1.fastq.gz, SRR8600103_1.fastq.gz, SRR8600105_1.fastq.gz, SRR8600110_1.fastq.gz, SRR8600113_1.fastq.gz, SRR8600117_1.fastq.gz, SRR8600121_1.fastq.gz, SRR8600122_1.fastq.gz, SRR8600123_1.fastq.gz, SRR8600128_1.fastq.gz, SRR8600133_1.fastq.gz, SRR8600134_1.fastq.gz, SRR8600136_1.fastq.gz, SRR8600145_1.fastq.gz, SRR8600147_1.fastq.gz, SRR8600160_1.fastq.gz, SRR8600164_1.fastq.gz, SRR8600166_1.fastq.gz, SRR8600167_1.fastq.gz, SRR8600174_1.fastq.gz, SRR8600182_1.fastq.gz, SRR8600184_1.fastq.gz, SRR8600185_1.fastq.gz, SRR8600188_1.fastq.gz, SRR8600189_1.fastq.gz, SRR8600190_1.fastq.gz, SRR8600193_1.fastq.gz, SRR8600194_1.fastq.gz, SRR8600216_1.fastq.gz, SRR8600222_1.fastq.gz, SRR8600223_1.fastq.gz, SRR8600226_1.fastq.gz, SRR8600228_1.fastq.gz, SRR8600235_1.fastq.gz, SRR8600236_1.fastq.gz, SRR8600243_1.fastq.gz, SRR8600244_1.fastq.gz, SRR8600251_1.fastq.gz, SRR8600252_1.fastq.gz, SRR8600258_1.fastq.gz, SRR8600259_1.fastq.gz, SRR8600264_1.fastq.gz, SRR8600267_1.fastq.gz, SRR8600268_1.fastq.gz, SRR8600270_1.fastq.gz, SRR8600272_1.fastq.gz, SRR8600274_1.fastq.gz, SRR8600277_1.fastq.gz, SRR8600278_1.fastq.gz, SRR8600281_1.fastq.gz, SRR8600282_1.fastq.gz, SRR8600283_1.fastq.gz, SRR8600285_1.fastq.gz, SRR8600287_1.fastq.gz, SRR8600288_1.fastq.gz, SRR8600291_1.fastq.gz, SRR8600294_1.fastq.gz, SRR8600296_1.fastq.gz, SRR8600302_1.fastq.gz, SRR8600308_1.fastq.gz, SRR8600309_1.fastq.gz, SRR8600311_1.fastq.gz, SRR8600319_1.fastq.gz, SRR8600327_1.fastq.gz, SRR8600328_1.fastq.gz, SRR8600329_1.fastq.gz, SRR8600331_1.fastq.gz, SRR8600339_1.fastq.gz, SRR8599995_2.fastq.gz, SRR8599997_2.fastq.gz, SRR8599998_2.fastq.gz, SRR8600006_2.fastq.gz, SRR8600007_2.fastq.gz, SRR8600008_2.fastq.gz, SRR8600014_2.fastq.gz, SRR8600015_2.fastq.gz, SRR8600020_2.fastq.gz, SRR8600022_2.fastq.gz, SRR8600026_2.fastq.gz, SRR8600031_2.fastq.gz, SRR8600044_2.fastq.gz, SRR8600045_2.fastq.gz, SRR8600048_2.fastq.gz, SRR8600049_2.fastq.gz, SRR8600052_2.fastq.gz, SRR8600055_2.fastq.gz, SRR8600058_2.fastq.gz, SRR8600059_2.fastq.gz, SRR8600064_2.fastq.gz, SRR8600070_2.fastq.gz, SRR8600072_2.fastq.gz, SRR8600075_2.fastq.gz, SRR8600079_2.fastq.gz, SRR8600092_2.fastq.gz, SRR8600093_2.fastq.gz, SRR8600098_2.fastq.gz, SRR8600100_2.fastq.gz, SRR8600101_2.fastq.gz, SRR8600102_2.fastq.gz, SRR8600103_2.fastq.gz, SRR8600105_2.fastq.gz, SRR8600110_2.fastq.gz, SRR8600113_2.fastq.gz, SRR8600117_2.fastq.gz, SRR8600121_2.fastq.gz, SRR8600122_2.fastq.gz, SRR8600123_2.fastq.gz, SRR8600128_2.fastq.gz, SRR8600133_2.fastq.gz, SRR8600134_2.fastq.gz, SRR8600136_2.fastq.gz, SRR8600145_2.fastq.gz, SRR8600147_2.fastq.gz, SRR8600160_2.fastq.gz, SRR8600164_2.fastq.gz, SRR8600166_2.fastq.gz, SRR8600167_2.fastq.gz, SRR8600174_2.fastq.gz, SRR8600182_2.fastq.gz, SRR8600184_2.fastq.gz, SRR8600185_2.fastq.gz, SRR8600188_2.fastq.gz, SRR8600189_2.fastq.gz, SRR8600190_2.fastq.gz, SRR8600193_2.fastq.gz, SRR8600194_2.fastq.gz, SRR8600216_2.fastq.gz, SRR8600222_2.fastq.gz, SRR8600223_2.fastq.gz, SRR8600226_2.fastq.gz, SRR8600228_2.fastq.gz, SRR8600235_2.fastq.gz, SRR8600236_2.fastq.gz, SRR8600243_2.fastq.gz, SRR8600244_2.fastq.gz, SRR8600251_2.fastq.gz, SRR8600252_2.fastq.gz, SRR8600258_2.fastq.gz, SRR8600259_2.fastq.gz, SRR8600264_2.fastq.gz, SRR8600267_2.fastq.gz, SRR8600268_2.fastq.gz, SRR8600270_2.fastq.gz, SRR8600272_2.fastq.gz, SRR8600274_2.fastq.gz, SRR8600277_2.fastq.gz, SRR8600278_2.fastq.gz, SRR8600281_2.fastq.gz, SRR8600282_2.fastq.gz, SRR8600283_2.fastq.gz, SRR8600285_2.fastq.gz, SRR8600287_2.fastq.gz, SRR8600288_2.fastq.gz, SRR8600291_2.fastq.gz, SRR8600294_2.fastq.gz, SRR8600296_2.fastq.gz, SRR8600302_2.fastq.gz, SRR8600308_2.fastq.gz, SRR8600309_2.fastq.gz, SRR8600311_2.fastq.gz, SRR8600319_2.fastq.gz, SRR8600327_2.fastq.gz, SRR8600328_2.fastq.gz, SRR8600329_2.fastq.gz, SRR8600331_2.fastq.gz, SRR8600339_2.fastq.gz 
  <span class="math-container">```</span>
  </code></pre>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-05-29T21:07:12.943Z
Comments.Text: This is easy. I'll respond over the weekend.
Comments.UserDisplayName: M__
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-06-08T08:25:46.983Z
Comments.Text: There is brand new spades release with a bunch of improvements. You may check if it is compatible with your HPC setup. https://github.com/ablab/spades/releases/tag/v4.0.0
Comments.UserDisplayName: darked89
Tags: <python><genome><hpc>
Title: Running multiple sequence using spades.py on an HPC
ThreadText: # Running multiple sequence using spades.py on an HPC
  
  (Asked by: username_0 on 2024-05-29)
  
  I am trying to run a batch script to assemble sequences using the spades.py command on HPC. I tested the script with one sequence (1 forward and 1 reverse read) and this worked. I have 100 sequences in total (200 files) and am trying to assemble all of them without having to resubmit the script each time. This is the original script that worked:
  
  ```
  #!/bin/bash  
   
  #SBATCH --job-name=pangenome_spades  
  #SBATCH --account=lilianasalvador  
  #SBATCH --partition=standard  
  #SBATCH --nodes=1  
  #SBATCH --ntasks=1  
  #SBATCH --mem-per-cpu=5gb  
  #SBATCH --time=01:00:00  
   
  spades.py -1 SRR8599992_1.fastq.gz -2 SRR8599992_2.fastq.gz -o /home/u32/username_0/spades_test -t 16 --careful 
  
  ```
  
  This is the script that I've been trying to run and the error it's giving me:
  
  ```
  #!/bin/bash  
   
  #SBATCH --job-name=pangenome_spades  
  #SBATCH --account=lilianasalvador  
  #SBATCH --partition=standard  
  #SBATCH --nodes=1  
  #SBATCH --ntasks=1  
  #SBATCH --mem-per-cpu=5gb  
  #SBATCH --time=02:00:00  
   
  module load python/3.9  
  module load spades/3.15.0-hpc 
  
  spades.py -1 *_1.fastq.gz -2 *_2.fastq.gz -o /groups/lilianasalvador/noah_100/SPAdes_output -t 16 --careful 
  
  SPAdes genome assembler v3.15.0-hpc 
   
  Usage: spades.py [options] -o <output_dir> 
  spades.py: error: Please specify option (e.g. -1, -2, -s, etc)) for the following paths: SRR8599995_1.fastq.gz, SRR8599997_1.fastq.gz, SRR8599998_1.fastq.gz, SRR8600006_1.fastq.gz, SRR8600007_1.fastq.gz, SRR8600008_1.fastq.gz, SRR8600014_1.fastq.gz, SRR8600015_1.fastq.gz, SRR8600020_1.fastq.gz, SRR8600022_1.fastq.gz, SRR8600026_1.fastq.gz, SRR8600031_1.fastq.gz, SRR8600044_1.fastq.gz, SRR8600045_1.fastq.gz, SRR8600048_1.fastq.gz, SRR8600049_1.fastq.gz, SRR8600052_1.fastq.gz, SRR8600055_1.fastq.gz, SRR8600058_1.fastq.gz, SRR8600059_1.fastq.gz, SRR8600064_1.fastq.gz, SRR8600070_1.fastq.gz, SRR8600072_1.fastq.gz, SRR8600075_1.fastq.gz, SRR8600079_1.fastq.gz, SRR8600092_1.fastq.gz, SRR8600093_1.fastq.gz, SRR8600098_1.fastq.gz, SRR8600100_1.fastq.gz, SRR8600101_1.fastq.gz, SRR8600102_1.fastq.gz, SRR8600103_1.fastq.gz, SRR8600105_1.fastq.gz, SRR8600110_1.fastq.gz, SRR8600113_1.fastq.gz, SRR8600117_1.fastq.gz, SRR8600121_1.fastq.gz, SRR8600122_1.fastq.gz, SRR8600123_1.fastq.gz, SRR8600128_1.fastq.gz, SRR8600133_1.fastq.gz, SRR8600134_1.fastq.gz, SRR8600136_1.fastq.gz, SRR8600145_1.fastq.gz, SRR8600147_1.fastq.gz, SRR8600160_1.fastq.gz, SRR8600164_1.fastq.gz, SRR8600166_1.fastq.gz, SRR8600167_1.fastq.gz, SRR8600174_1.fastq.gz, SRR8600182_1.fastq.gz, SRR8600184_1.fastq.gz, SRR8600185_1.fastq.gz, SRR8600188_1.fastq.gz, SRR8600189_1.fastq.gz, SRR8600190_1.fastq.gz, SRR8600193_1.fastq.gz, SRR8600194_1.fastq.gz, SRR8600216_1.fastq.gz, SRR8600222_1.fastq.gz, SRR8600223_1.fastq.gz, SRR8600226_1.fastq.gz, SRR8600228_1.fastq.gz, SRR8600235_1.fastq.gz, SRR8600236_1.fastq.gz, SRR8600243_1.fastq.gz, SRR8600244_1.fastq.gz, SRR8600251_1.fastq.gz, SRR8600252_1.fastq.gz, SRR8600258_1.fastq.gz, SRR8600259_1.fastq.gz, SRR8600264_1.fastq.gz, SRR8600267_1.fastq.gz, SRR8600268_1.fastq.gz, SRR8600270_1.fastq.gz, SRR8600272_1.fastq.gz, SRR8600274_1.fastq.gz, SRR8600277_1.fastq.gz, SRR8600278_1.fastq.gz, SRR8600281_1.fastq.gz, SRR8600282_1.fastq.gz, SRR8600283_1.fastq.gz, SRR8600285_1.fastq.gz, SRR8600287_1.fastq.gz, SRR8600288_1.fastq.gz, SRR8600291_1.fastq.gz, SRR8600294_1.fastq.gz, SRR8600296_1.fastq.gz, SRR8600302_1.fastq.gz, SRR8600308_1.fastq.gz, SRR8600309_1.fastq.gz, SRR8600311_1.fastq.gz, SRR8600319_1.fastq.gz, SRR8600327_1.fastq.gz, SRR8600328_1.fastq.gz, SRR8600329_1.fastq.gz, SRR8600331_1.fastq.gz, SRR8600339_1.fastq.gz, SRR8599995_2.fastq.gz, SRR8599997_2.fastq.gz, SRR8599998_2.fastq.gz, SRR8600006_2.fastq.gz, SRR8600007_2.fastq.gz, SRR8600008_2.fastq.gz, SRR8600014_2.fastq.gz, SRR8600015_2.fastq.gz, SRR8600020_2.fastq.gz, SRR8600022_2.fastq.gz, SRR8600026_2.fastq.gz, SRR8600031_2.fastq.gz, SRR8600044_2.fastq.gz, SRR8600045_2.fastq.gz, SRR8600048_2.fastq.gz, SRR8600049_2.fastq.gz, SRR8600052_2.fastq.gz, SRR8600055_2.fastq.gz, SRR8600058_2.fastq.gz, SRR8600059_2.fastq.gz, SRR8600064_2.fastq.gz, SRR8600070_2.fastq.gz, SRR8600072_2.fastq.gz, SRR8600075_2.fastq.gz, SRR8600079_2.fastq.gz, SRR8600092_2.fastq.gz, SRR8600093_2.fastq.gz, SRR8600098_2.fastq.gz, SRR8600100_2.fastq.gz, SRR8600101_2.fastq.gz, SRR8600102_2.fastq.gz, SRR8600103_2.fastq.gz, SRR8600105_2.fastq.gz, SRR8600110_2.fastq.gz, SRR8600113_2.fastq.gz, SRR8600117_2.fastq.gz, SRR8600121_2.fastq.gz, SRR8600122_2.fastq.gz, SRR8600123_2.fastq.gz, SRR8600128_2.fastq.gz, SRR8600133_2.fastq.gz, SRR8600134_2.fastq.gz, SRR8600136_2.fastq.gz, SRR8600145_2.fastq.gz, SRR8600147_2.fastq.gz, SRR8600160_2.fastq.gz, SRR8600164_2.fastq.gz, SRR8600166_2.fastq.gz, SRR8600167_2.fastq.gz, SRR8600174_2.fastq.gz, SRR8600182_2.fastq.gz, SRR8600184_2.fastq.gz, SRR8600185_2.fastq.gz, SRR8600188_2.fastq.gz, SRR8600189_2.fastq.gz, SRR8600190_2.fastq.gz, SRR8600193_2.fastq.gz, SRR8600194_2.fastq.gz, SRR8600216_2.fastq.gz, SRR8600222_2.fastq.gz, SRR8600223_2.fastq.gz, SRR8600226_2.fastq.gz, SRR8600228_2.fastq.gz, SRR8600235_2.fastq.gz, SRR8600236_2.fastq.gz, SRR8600243_2.fastq.gz, SRR8600244_2.fastq.gz, SRR8600251_2.fastq.gz, SRR8600252_2.fastq.gz, SRR8600258_2.fastq.gz, SRR8600259_2.fastq.gz, SRR8600264_2.fastq.gz, SRR8600267_2.fastq.gz, SRR8600268_2.fastq.gz, SRR8600270_2.fastq.gz, SRR8600272_2.fastq.gz, SRR8600274_2.fastq.gz, SRR8600277_2.fastq.gz, SRR8600278_2.fastq.gz, SRR8600281_2.fastq.gz, SRR8600282_2.fastq.gz, SRR8600283_2.fastq.gz, SRR8600285_2.fastq.gz, SRR8600287_2.fastq.gz, SRR8600288_2.fastq.gz, SRR8600291_2.fastq.gz, SRR8600294_2.fastq.gz, SRR8600296_2.fastq.gz, SRR8600302_2.fastq.gz, SRR8600308_2.fastq.gz, SRR8600309_2.fastq.gz, SRR8600311_2.fastq.gz, SRR8600319_2.fastq.gz, SRR8600327_2.fastq.gz, SRR8600328_2.fastq.gz, SRR8600329_2.fastq.gz, SRR8600331_2.fastq.gz, SRR8600339_2.fastq.gz 
  ```
  
  ```
  
  **Tags:** python, genome, hpc
  
  > **Comments:**
  > **username_1**: This is easy. I'll respond over the weekend.  
  > **username_2**: There is brand new spades release with a bunch of improvements. You may check if it is compatible with your HPC setup. https://github.com/ablab/spades/releases/tag/v4.0.0  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_3 on 2024-05-30. Score: 5
  
  I have some time now, so I'm going to take a shot at this, @M\_\_
  
  You need to run spades per read pair. When you specify `*_1.fastq.gz` as the `-1` param, you're giving it *every* `_1` file at once, not one `_1` file per spades command.
  
  I'd recommend this:
  
  ```
  for smpl in $(ls *_1.fastq.gz | xargs -I infile basename infile _1.fastq.gz)
  do
      cat > "./${smpl}_spades.slurm" <<EOF
  #!/bin/bash  
   
  #SBATCH --job-name=pangenome_spades.${smpl}  
  #SBATCH --account=lilianasalvador
  #SBATCH --partition=standard
  #SBATCH --nodes=1
  #SBATCH --ntasks=1
  #SBATCH --mem-per-cpu=5gb
  #SBATCH --time=02:00:00
  
  spades.py -1 ${smpl}_1.fastq.gz -2 ${smpl}_2.fastq.gz -o groups/lilianasalvador/noah_100/SPAdes_output_${smpl} -t 16 --careful
  EOF
  done
  
  ```
  
  Run a sanity check on a few of the `.slurm` files, then
  
  ```
  ls *.slurm | xargs -I job_file sbatch job_file
  
  ```
  
  > **Comments:**
  > **username_1**: Cool, thanks. I was going to solve this via a formal Slurm array job, but this will work.  
  > **username_3**: Please do add an array-based answer, I'm not familiar with arrays but would love to learn  
  > **username_0**: Thank you! This worked but we're only getting one contigs.fasta file when we run. Should we be getting a contigs.fasta per sequence pair or one for all the sequences we ran?  
  > **username_3**: I am not familiar with spades but going by M__'s answer, you need something at the `-o` param as well. I'll edit my answer but you should test running 2 samples at the same time before running all of the scripts concurrently.  
  
  ---
  
  ### Answer by username_1 on 2024-06-01. Score: 3
  
  The *job array* is a formal approach to solving this via SLURM
  
  What I am not clear about is the number of files `SRR8599992_1.fastq.gz` `SRR8599992_2.fastq.gaz` I originally thought the final number of files was around 100, I've guessed at 120
  
  ```
  #SBATCH --job-name=pangenome_spade
  #SBATCH --account=lilianasalvador
  #SBATCH --ntasks=120
  #SBATCH --cpus-per-task=16 
  #SBATCH --mem-per-cpu=5gb  
  #SBATCH --time=02:00:00 
  #SBATCH --array=599992-5600100
  
  module load python/3.9  
  module load spades/3.15.0-hpc 
  
  spades.py -1 'SRR'${SLURM_ARRAY_TASK_ID}'_1.fastq.gz' -2 'SRR'${$SLURM_ARRAY_TASK_ID}'_2.fastq.gz' -o /groups/lilianasalvador/noah_100/'SPAdes_output_'${SLURM_ARRAY_TASK_ID}'.fastq' -t 16 --careful 
  
  ```
  
  I would test this via an `echo` test script writing to a specific file to make sure the `spades.py` command produces the correct command line for each iteration.
  
  Things to note ...
  
  * Array jobs is specified by the `$SLURM_ARRAY_TASK_ID` variable and `#SBATCH --array=599992-5600100`
  * `ntasks` is the total number of spades assemblies you are performing here I guessed at 120
  * You are parallelising via `-t 16` therefore this must be reflected in your submission job via `--cpus-per-task`
  * The *ouput* file *MUST* use `$SLURM_ARRAY_TASK_ID` otherwise you will continually override your out file.
  
  ---
  
  This is not an optimal SLURM job because ...
  
  * `-t 16` you need to check with your HPC engineer, this might take a long time to get through the queue and an alternative architecture to permit 16 CPUs can get onto the cluster. HTC users will jump ahead of you, so on a hypervisor setup you can specify a separate cluster for your work. This is not well explained but hope it makes sense.
  * Shifting to high throughput computing (HTC) could be considered here to get this through a lot quicker, e.g. this is 120 sequential assemblies, 10 sequential assemblies for 12 separate submissions would be a lot quicker (you just alter the "array job" inputs range.
  
  
  ---
  
  