Answers.Body: <p>I see two probable sources of error. The general format of the bwa command you are trying to run is</p>
  <pre><code>bwa mem genome.fa read1.fastq.gz read2.fastq.gz 
  </code></pre>
  <p>However, you are running (with some options removed and paths truncated for brevity):</p>
  <pre><code>bwa mem *_contigs.fasta /groups/[...]/SRR* -r *_1.[...].fastq.gz *_2.[..].fastq.gz 
  </code></pre>
  <p>Now, I don't know what the <a href="https://en.wikipedia.org/wiki/Glob_(programming)" rel="nofollow noreferrer">globs</a> <code>*_contigs.fasta</code>, <code>/groups/[...]/SRR*</code>, <code>*_1.[...].fastq.gz</code> or <code>*_2.[..].fastq.gz</code> would expand to but I doubt the result of expanding so many globs will be just one genome file and two fastq files. In addition, you seem to be already running this once for every file ending in <code>_contigs.fasta</code> since you have it in the loop:</p>
  <pre><code>for file in *_contigs.fasta; do ...; done
  </code></pre>
  <p>Assuming you actually have <em>one</em> file wth all of your contigs and this file has been indexed and that's what you want to use as a genome, then you would want</p>
  <pre><code>bwa mem /path/to/contig/contigs.fasta read1.fastq.gz read2.fastq.gz
  </code></pre>
  <p>I don't quite understand what you have from your question though so I can't be more specific. Next, presumably you have various sequencing run outputs, so multiple pairs of fastq files. Those are the ones your want to iterate over, not the contigs. So you would want a loop like this (I removed the unnecessary <code>samtools view</code> step, since <code>samtools sort</code> can handle the sam to bam conversion, and the also unnecessary <code>-r</code> option; if you <em>know</em> you need the <code>-r</code>, you can put it back):</p>
  <pre class="lang-sh prettyprint-override"><code>outBamsDir=/groups/lilianasalvador/noah_100/bwa/bwa_mem_output/
  for file in *_1.fastq.00.0_0.cor.fastq.gz; do  
      sample_name=${file%_1*}
      file2=${file/_1/_2}  
      mkdir -p &quot;$outBamsDir&quot;/&quot;$sample_name&quot;
      bwa mem \ 
        -t 14 \
        /path/to/contig/contigs.fasta \
        &quot;$file&quot; &quot;$file2&quot; | 
           samtools sort -@14 \
            -o &quot;$outBamsDir/$sample_name/$sample_name&quot;.bam
  done
  </code></pre>
  <p>Or, if you really have multiple contig files and want to treat each of them as the reference genome to align against, so presumably you only have 2 fastq files, then the loop would be:</p>
  <pre class="lang-sh prettyprint-override"><code>outBamsDir=/groups/lilianasalvador/noah_100/bwa/bwa_mem_output/
  for file in *_contigs.fasta; do  
    sample_name=${file%_contigs.fasta}  
    mkdir -p &quot;$outBamsDir&quot;/&quot;$sample_name&quot;
    bwa mem -t 14 /whatever/you/want/to/use/as/genome.fa \
     &quot;$file&quot; | 
     samtools sort -@14 -o &quot;$outBamsDir/$sample_name&quot;.bam 
  done
  </code></pre>
  <p>That, of course, assumes single-end sequencing with one file per sample.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-07-01T09:41:56.113Z
Answers.Text: Good points. AFAIK neither `bwa`  nor `bwa-mem2` can take more than 2 FASTQ files, guess the paired or non-paired mode and align them. Besides, using a single computing node to align multiple FASTQ pairs sequentially is not optimal save for a single node "cluster".
Answers.UserDisplayName: darked89
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-07-01T21:21:02.040Z
Answers.Text: I have a contigs.fasta file for each sequence as well as two .cor.fastq.gz files (reverse and forward) for each sequence. In /groups/lilianasalvador/noah_100/bwa/bwa_output I have 100 files - each corresponding to a sequence (i.e. SRR8599992, SRR8599993...) and within each of those is the contigs.fasta and the .cor.fastq.gz files for the sequence. That's why I was saying *_contigs.fasta /groups/lilianasalvador/noah_100/bwa/bwa_output/SRR* to try to explain that the contigs.fasta files are within the corresponding directory that starts with SRR. Idk how to correctly write the path
Answers.UserDisplayName: Lucy Kelly
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-07-01T21:22:10.890Z
Answers.Text: Do I have too many files? I was told I should have a contigs.fasta file for every sequence, which is why they're separated in directories
Answers.UserDisplayName: Lucy Kelly
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-07-02T14:09:12.510Z
Answers.Text: You might or you.might not, but we cannot know since you never showed us the files or even explained what you are really trying to do. However if you found a solution, please post that as an answer and not as part of the question.
Answers.UserDisplayName: terdon
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-06-29T12:31:22.137Z
Answers.LastActivityDate: 2024-07-01T09:49:19.093Z
Answers.LastEditDate: 2024-07-01T09:49:19.093Z
Answers.UserDisplayName: terdon
Answers.Body: <p>I note that you make a mistake in your command lines,it seems that you want to map reads to the ref , convert the sam to the bam file and sort the bam file in one command, actually, you need to get the unsorted bam file first and sort the bam file with samtools singly. Because <code>samtools sort</code> will sort the whole file,but if you use the pipe as input of the 'samtools sort', the result is not a whole file from the <code>samtools view -Sb</code>,it just is a part of the resulty,so the <code>samtools sort</code> can't sort it , and the result may be stored in the memory , when many parts of results into the memory,it will be killed as OOM. You can try to map and convert sam to bam first and do the sorting singly.</p>
  <p>I find that it contains <code>/groups/lilianasalvador/noah_100/bwa/bwa_output/SRR</code> between the reference sequence and the <code>-r</code> , it seems don't need this parameter , you can confirm the manual of the bwa mem</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-06-29T11:33:03.727Z
Answers.Text: This isn't true. Samtools sort is perfectly capable of sorting from a pipe. Try it, just run `samtools view -h foo.bam | samtools sort -o bar.bam`. And it can do this with humongous files, it doesn't store in memory, it creates temporary files while sorting precisely to avoid the problem you mention.
Answers.UserDisplayName: terdon
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-06-30T05:53:13.467Z
Answers.Text: Sorry about that,actually i have tried many times in my machine, but that's not hpc,it just has 64GB memory, and I indeed got that problem,when i use the bwa mem | samtools view | samtools sort my computer will be down, i will can't connect it by ssh, and after long time ,it will recover,but that is uncontrollable ,and i hava find out some oom events during the analysis when i can connect it again. I have resolved it by the way i hava said.But there is an other question in his commandline which has been mentioned in the second paragraph in my answer last post.It seems that you mention it too.
Answers.UserDisplayName: kkmonster
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-06-30T14:59:07.893Z
Answers.Text: That can be caused by various things, but I can assure you, I have been running `bwa mem | samtools sort` in a professional, production pipeline for almost a decade now. And also did so on my laptop, before posting, just to double check. I also confirmed in the [samtools manual\(https://www.htslib.org/doc/samtools-sort.html) that it creates temp files when memory is not enough. As for the last paragraph, I'm sorry but I don't understand it at all. Yes, the `-r` isn't needed, but it isn't causing the problem either.
Answers.UserDisplayName: terdon
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-07-01T01:28:28.813Z
Answers.Text: Thanks to reply, actually i have noted the directory which he should not use between the reference and the reads file too. But the words may be unclear.  On the other hand i major in computer science ,when i get the  problem i have mentioned and take account for the behaviour of my comuputer and the syslog, i think i could be the reason what i have said , and by split the command line i resolve it. So it makes me believe the reason more.  The version of the samtools and htslib is 1.20. I will try to do reproduction experiments to located the ture reason. Thank you for reminding me.
Answers.UserDisplayName: kkmonster
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-06-29T08:45:02.570Z
Answers.LastActivityDate: 2024-06-29T08:45:02.570Z
Answers.UserDisplayName: kkmonster
Body: <p>I am trying to write a script that will run this line from <a href="https://github.com/kmceres/Mbovis_pangenome/blob/main/Parameter_file.md" rel="nofollow noreferrer">https://github.com/kmceres/Mbovis_pangenome/blob/main/Parameter_file.md</a>:</p>
  <pre><code>bwa mem -t 14 /path/to/contig/contigs.fasta -r contig_1P.fq.gz contig_2P.fq.gz | 
     samtools view -Sb | 
        samtools sort -@14 -o /path/to/contig/contig.sorted.bam
  </code></pre>
  <p>All of the files are in directories under their sequence number within a <code>bwa_output</code> (from when I ran bwa index) directory (e.g. <code>/groups/lilianasalvador/noah_100/bwa/bwa_output/SRR8599992</code>)</p>
  <p>Here is my script:</p>
  <pre><code>#!/bin/bash     
  
  #SBATCH --job-name=bwa_mem     
  #SBATCH --account=lilianasalvador   
  #SBATCH --partition=standard   
  #SBATCH --nodes=1   
  #SBATCH --ntasks=1   
  #SBATCH --mem-per-cpu=16gb   
  #SBATCH --time=99:00:00   
  
  module load bwa/0.7.17    
  module load samtools/1.19.2 
  
  for file in *_contigs.fasta; do  
      sample_name=<span class="math-container">${file%_contigs.fasta}  
  mkdir -p /groups/lilianasalvador/noah_100/bwa/bwa_mem_output/$</span>sample_name   
  
  bwa mem -t 14 *_contigs.fasta /groups/lilianasalvador/noah_100/bwa/bwa_output/SRR* -r *_1.fastq.00.0_0.cor.fastq.gz *_2.fastq.00.0_0.cor.fastq.gz | samtools view -Sb | samtools sort -@14 -o /groups/lilianasalvador/noah_100/bwa/bwa_mem_output 
  
  done
  </code></pre>
  <p>I'm getting this error:</p>
  <pre><code>Usage: bwa mem [options] &lt;idxbase&gt; &lt;in1.fq&gt; [in2.fq] 
  ...
  [main_samview] fail to read the header from &quot;-&quot;. 
  [W::hts_set_opt] Cannot change block size for this format 
  samtools sort: failed to read header from &quot;-&quot; 
  </code></pre>
  <p>I am relatively new to all of this and would appreciate some help editing my script</p>
  <p>update 7/1: I tried running this</p>
  <pre><code>module load bwa/0.7.17    
  module load samtools/1.19.2 
  
  outBamsDir=/groups/lilianasalvador/noah_100/bwa/bwa_mem_output/ 
  
  for file in *_1.fastq.00.0_0.cor.fastq.gz; do   
      sample_name=<span class="math-container">${file%_1*} 
  file2=$</span>{file/_1/_2}   
      mkdir -p &quot;<span class="math-container">$outBamsDir"/"$</span>sample_name&quot; 
      bwa mem \  
        -t 14 \ 
  /groups/lilianasalvador/noah_100/bwa/bwa_output/SRR8599992/SRR8599992_contigs.fasta \ 
        &quot;<span class="math-container">$file" "$</span>file2&quot; |  
           samtools sort -@14 \ 
            -o &quot;<span class="math-container">$outBamsDir/$</span>sample_name/$sample_name&quot;.bam 
  done
  </code></pre>
  <p>and I got this error</p>
  <pre><code>/var/spool/slurm/d/job10075741/slurm_script: line 40: -t: command not found 
  /var/spool/slurm/d/job10075741/slurm_script: line 42: /groups/lilianasalvador/noah_100/bwa/bwa_output/SRR8599992/SRR8599992_contigs.fasta: Permission denied 
  /var/spool/slurm/d/job10075741/slurm_script: line 44: SRR8599992_1.fastq.00.0_0.cor.fastq.gz: command not found 
  [E::hts_open_format] Failed to open file &quot; &quot; : No such file or directory 
  samtools sort: can't open &quot; &quot;: No such file or directory 
  /var/spool/slurm/d/job10075741/slurm_script: line 48: -o: command not found 
  </code></pre>
  <p>I also tried this:</p>
  <pre><code>module load bwa/0.7.17    
  module load samtools/1.19.2 
  
  for file in *_contigs.fasta; do  
      sample_name=<span class="math-container">${file%_contigs.fasta}  
  mkdir -p /groups/lilianasalvador/noah_100/bwa/bwa_mem_output/$</span>sample_name   
  
      bwa mem -t 14 /groups/lilianasalvador/noah_100/bwa/bwa_output/SRR8599992/SRR8599992_contigs.fasta -r SRR8599992_1.fastq.00.0_0.cor.fastq.gz SRR8599992_2.fastq.00.1_0.cor.fastq.gz | samtools view -Sb | samtools sort -@14 -o /groups/lilianasalvador/noah_100/bwa/bwa_mem_output/SRR8599992 
  
  done
  </code></pre>
  <p>and got this error</p>
  <pre><code>[M::bwa_idx_load_from_disk] read 0 ALT contigs 
  [M::process] read 648167 sequences (95550856 bp)... 
  [M::mem_process_seqs] Processed 648167 reads in 28.660 CPU sec, 28.709 real sec 
  [main] Version: 0.7.17-r1188 
  [main] CMD: bwa mem -t 14 -r SRR8599992_1.fastq.00.0_0.cor.fastq.gz /groups/lilianasalvador/noah_100/bwa/bwa_output/SRR8599992/SRR8599992_contigs.fasta SRR8599992_2.fastq.00.1_0.cor.fastq.gz 
  [main] Real time: 42.903 sec; CPU: 30.392 sec 
  [bam_sort_core] merging from 0 files and 14 in-memory blocks... 
  [E::hts_open_format] Failed to open file &quot;/groups/lilianasalvador/noah_100/bwa/bwa_mem_output&quot; : Is a directory 
  samtools sort: failed to create &quot;/groups/lilianasalvador/noah_100/bwa/bwa_mem_output&quot;: Is a directory 
  </code></pre>
  <p>This is what worked:</p>
  <pre><code>module load bwa/0.7.17    
  module load samtools/1.19.2 
  
  bwa mem -t 14 /groups/lilianasalvador/noah_100/bwa/bwa_output/SRR8599992/SRR8599992_contigs.fasta -r SRR8599992_1.fastq.00.0_0.cor.fastq.gz SRR8599992_2.fastq.00.1_0.cor.fastq.gz | 
     samtools view -Sb | 
      samtools sort -@14 -o /groups/lilianasalvador/noah_100/bwa/bwa_mem_output/SRR8599992/SRR8599992.contig.sorted.bam
  
  </code></pre>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-06-29T11:52:05.753Z
Comments.Text: What exactly are you trying to do? Do you have _one_ set of fastq files and you want to align that against multiple genomes (contigs) or do you have many sets of fastq files, the output of multiple sequencing runs (or the same run but multiple sequenced samples) and you want to align each set of fastqs against the contigs?
Comments.UserDisplayName: terdon
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-06-29T11:54:10.010Z
Comments.Text: Please [edit] your question and explain what your files are, and then also show us the output of `ls *_contigs.fasta`, `ls /groups/lilianasalvador/noah_100/bwa/bwa_output/SRR*`, `ls *_1.fastq.00.0_0.cor.fastq.gz` and `ls  *_2.fastq.00.0_0.cor.fastq.gz`. I tried to answer but without knowing what the actual data and end objective are, I can only make guesses.
Comments.UserDisplayName: terdon
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-07-02T10:16:51.220Z
Comments.Text: Please read https://bioinformatics.stackexchange.com/help/formatting and format your code as code.
Comments.UserDisplayName: terdon
Tags: <samtools><bwa><hpc>
Title: Running bwa mem and samtools in one script on a linux cluster
ThreadText: # Running bwa mem and samtools in one script on a linux cluster
  
  (Asked by: username_0 on 2024-06-28)
  
  I am trying to write a script that will run this line from <https://github.com/kmceres/Mbovis_pangenome/blob/main/Parameter_file.md>:
  
  ```
  bwa mem -t 14 /path/to/contig/contigs.fasta -r contig_1P.fq.gz contig_2P.fq.gz | 
     samtools view -Sb | 
        samtools sort -@14 -o /path/to/contig/contig.sorted.bam
  
  ```
  
  All of the files are in directories under their sequence number within a `bwa_output` (from when I ran bwa index) directory (e.g. `/groups/lilianasalvador/noah_100/bwa/bwa_output/SRR8599992`)
  
  Here is my script:
  
  ```
  #!/bin/bash     
  
  #SBATCH --job-name=bwa_mem     
  #SBATCH --account=lilianasalvador   
  #SBATCH --partition=standard   
  #SBATCH --nodes=1   
  #SBATCH --ntasks=1   
  #SBATCH --mem-per-cpu=16gb   
  #SBATCH --time=99:00:00   
  
  module load bwa/0.7.17    
  module load samtools/1.19.2 
  
  for file in *_contigs.fasta; do  
      sample_name=${file%_contigs.fasta}  
  mkdir -p /groups/lilianasalvador/noah_100/bwa/bwa_mem_output/$sample_name   
  
  bwa mem -t 14 *_contigs.fasta /groups/lilianasalvador/noah_100/bwa/bwa_output/SRR* -r *_1.fastq.00.0_0.cor.fastq.gz *_2.fastq.00.0_0.cor.fastq.gz | samtools view -Sb | samtools sort -@14 -o /groups/lilianasalvador/noah_100/bwa/bwa_mem_output 
  
  done
  
  ```
  
  I'm getting this error:
  
  ```
  Usage: bwa mem [options] <idxbase> <in1.fq> [in2.fq] 
  ...
  [main_samview] fail to read the header from "-". 
  [W::hts_set_opt] Cannot change block size for this format 
  samtools sort: failed to read header from "-" 
  
  ```
  
  I am relatively new to all of this and would appreciate some help editing my script
  
  update 7/1: I tried running this
  
  ```
  module load bwa/0.7.17    
  module load samtools/1.19.2 
  
  outBamsDir=/groups/lilianasalvador/noah_100/bwa/bwa_mem_output/ 
  
  for file in *_1.fastq.00.0_0.cor.fastq.gz; do   
      sample_name=${file%_1*} 
  file2=${file/_1/_2}   
      mkdir -p "$outBamsDir"/"$sample_name" 
      bwa mem \  
        -t 14 \ 
  /groups/lilianasalvador/noah_100/bwa/bwa_output/SRR8599992/SRR8599992_contigs.fasta \ 
        "$file" "$file2" |  
           samtools sort -@14 \ 
            -o "$outBamsDir/$sample_name/$sample_name".bam 
  done
  
  ```
  
  and I got this error
  
  ```
  /var/spool/slurm/d/job10075741/slurm_script: line 40: -t: command not found 
  /var/spool/slurm/d/job10075741/slurm_script: line 42: /groups/lilianasalvador/noah_100/bwa/bwa_output/SRR8599992/SRR8599992_contigs.fasta: Permission denied 
  /var/spool/slurm/d/job10075741/slurm_script: line 44: SRR8599992_1.fastq.00.0_0.cor.fastq.gz: command not found 
  [E::hts_open_format] Failed to open file " " : No such file or directory 
  samtools sort: can't open " ": No such file or directory 
  /var/spool/slurm/d/job10075741/slurm_script: line 48: -o: command not found 
  
  ```
  
  I also tried this:
  
  ```
  module load bwa/0.7.17    
  module load samtools/1.19.2 
  
  for file in *_contigs.fasta; do  
      sample_name=${file%_contigs.fasta}  
  mkdir -p /groups/lilianasalvador/noah_100/bwa/bwa_mem_output/$sample_name   
  
      bwa mem -t 14 /groups/lilianasalvador/noah_100/bwa/bwa_output/SRR8599992/SRR8599992_contigs.fasta -r SRR8599992_1.fastq.00.0_0.cor.fastq.gz SRR8599992_2.fastq.00.1_0.cor.fastq.gz | samtools view -Sb | samtools sort -@14 -o /groups/lilianasalvador/noah_100/bwa/bwa_mem_output/SRR8599992 
  
  done
  
  ```
  
  and got this error
  
  ```
  [M::bwa_idx_load_from_disk] read 0 ALT contigs 
  [M::process] read 648167 sequences (95550856 bp)... 
  [M::mem_process_seqs] Processed 648167 reads in 28.660 CPU sec, 28.709 real sec 
  [main] Version: 0.7.17-r1188 
  [main] CMD: bwa mem -t 14 -r SRR8599992_1.fastq.00.0_0.cor.fastq.gz /groups/lilianasalvador/noah_100/bwa/bwa_output/SRR8599992/SRR8599992_contigs.fasta SRR8599992_2.fastq.00.1_0.cor.fastq.gz 
  [main] Real time: 42.903 sec; CPU: 30.392 sec 
  [bam_sort_core] merging from 0 files and 14 in-memory blocks... 
  [E::hts_open_format] Failed to open file "/groups/lilianasalvador/noah_100/bwa/bwa_mem_output" : Is a directory 
  samtools sort: failed to create "/groups/lilianasalvador/noah_100/bwa/bwa_mem_output": Is a directory 
  
  ```
  
  This is what worked:
  
  ```
  module load bwa/0.7.17    
  module load samtools/1.19.2 
  
  bwa mem -t 14 /groups/lilianasalvador/noah_100/bwa/bwa_output/SRR8599992/SRR8599992_contigs.fasta -r SRR8599992_1.fastq.00.0_0.cor.fastq.gz SRR8599992_2.fastq.00.1_0.cor.fastq.gz | 
     samtools view -Sb | 
      samtools sort -@14 -o /groups/lilianasalvador/noah_100/bwa/bwa_mem_output/SRR8599992/SRR8599992.contig.sorted.bam
  
  ```
  
  **Tags:** samtools, bwa, hpc
  
  > **Comments:**
  > **username_1**: What exactly are you trying to do? Do you have _one_ set of fastq files and you want to align that against multiple genomes (contigs) or do you have many sets of fastq files, the output of multiple sequencing runs (or the same run but multiple sequenced samples) and you want to align each set of fastqs against the contigs?  
  > **username_1**: Please [edit] your question and explain what your files are, and then also show us the output of `ls *_contigs.fasta`, `ls /groups/lilianasalvador/noah_100/bwa/bwa_output/SRR*`, `ls *_1.fastq.00.0_0.cor.fastq.gz` and `ls  *_2.fastq.00.0_0.cor.fastq.gz`. I tried to answer but without knowing what the actual data and end objective are, I can only make guesses.  
  > **username_1**: Please read https://bioinformatics.stackexchange.com/help/formatting and format your code as code.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2024-06-29. Score: 3
  
  I see two probable sources of error. The general format of the bwa command you are trying to run is
  
  ```
  bwa mem genome.fa read1.fastq.gz read2.fastq.gz 
  
  ```
  
  However, you are running (with some options removed and paths truncated for brevity):
  
  ```
  bwa mem *_contigs.fasta /groups/[...]/SRR* -r *_1.[...].fastq.gz *_2.[..].fastq.gz 
  
  ```
  
  Now, I don't know what the [globs](https://en.wikipedia.org/wiki/Glob_(programming)) `*_contigs.fasta`, `/groups/[...]/SRR*`, `*_1.[...].fastq.gz` or `*_2.[..].fastq.gz` would expand to but I doubt the result of expanding so many globs will be just one genome file and two fastq files. In addition, you seem to be already running this once for every file ending in `_contigs.fasta` since you have it in the loop:
  
  ```
  for file in *_contigs.fasta; do ...; done
  
  ```
  
  Assuming you actually have *one* file wth all of your contigs and this file has been indexed and that's what you want to use as a genome, then you would want
  
  ```
  bwa mem /path/to/contig/contigs.fasta read1.fastq.gz read2.fastq.gz
  
  ```
  
  I don't quite understand what you have from your question though so I can't be more specific. Next, presumably you have various sequencing run outputs, so multiple pairs of fastq files. Those are the ones your want to iterate over, not the contigs. So you would want a loop like this (I removed the unnecessary `samtools view` step, since `samtools sort` can handle the sam to bam conversion, and the also unnecessary `-r` option; if you *know* you need the `-r`, you can put it back):
  
  ```sh
  outBamsDir=/groups/lilianasalvador/noah_100/bwa/bwa_mem_output/
  for file in *_1.fastq.00.0_0.cor.fastq.gz; do  
      sample_name=${file%_1*}
      file2=${file/_1/_2}  
      mkdir -p "$outBamsDir"/"$sample_name"
      bwa mem \ 
        -t 14 \
        /path/to/contig/contigs.fasta \
        "$file" "$file2" | 
           samtools sort -@14 \
            -o "$outBamsDir/$sample_name/$sample_name".bam
  done
  
  ```
  
  Or, if you really have multiple contig files and want to treat each of them as the reference genome to align against, so presumably you only have 2 fastq files, then the loop would be:
  
  ```sh
  outBamsDir=/groups/lilianasalvador/noah_100/bwa/bwa_mem_output/
  for file in *_contigs.fasta; do  
    sample_name=${file%_contigs.fasta}  
    mkdir -p "$outBamsDir"/"$sample_name"
    bwa mem -t 14 /whatever/you/want/to/use/as/genome.fa \
     "$file" | 
     samtools sort -@14 -o "$outBamsDir/$sample_name".bam 
  done
  
  ```
  
  That, of course, assumes single-end sequencing with one file per sample.
  
  > **Comments:**
  > **username_2**: Good points. AFAIK neither `bwa`  nor `bwa-mem2` can take more than 2 FASTQ files, guess the paired or non-paired mode and align them. Besides, using a single computing node to align multiple FASTQ pairs sequentially is not optimal save for a single node "cluster".  
  > **username_0**: I have a contigs.fasta file for each sequence as well as two .cor.fastq.gz files (reverse and forward) for each sequence. In /groups/lilianasalvador/noah_100/bwa/bwa_output I have 100 files - each corresponding to a sequence (i.e. SRR8599992, SRR8599993...) and within each of those is the contigs.fasta and the .cor.fastq.gz files for the sequence. That's why I was saying *_contigs.fasta /groups/lilianasalvador/noah_100/bwa/bwa_output/SRR* to try to explain that the contigs.fasta files are within the corresponding directory that starts with SRR. Idk how to correctly write the path  
  > **username_0**: Do I have too many files? I was told I should have a contigs.fasta file for every sequence, which is why they're separated in directories  
  > **username_1**: You might or you.might not, but we cannot know since you never showed us the files or even explained what you are really trying to do. However if you found a solution, please post that as an answer and not as part of the question.  
  
  ---
  
  ### Answer by username_3 on 2024-06-29. Score: -2
  
  I note that you make a mistake in your command lines,it seems that you want to map reads to the ref , convert the sam to the bam file and sort the bam file in one command, actually, you need to get the unsorted bam file first and sort the bam file with samtools singly. Because `samtools sort` will sort the whole file,but if you use the pipe as input of the 'samtools sort', the result is not a whole file from the `samtools view -Sb`,it just is a part of the resulty,so the `samtools sort` can't sort it , and the result may be stored in the memory , when many parts of results into the memory,it will be killed as OOM. You can try to map and convert sam to bam first and do the sorting singly.
  
  I find that it contains `/groups/lilianasalvador/noah_100/bwa/bwa_output/SRR` between the reference sequence and the `-r` , it seems don't need this parameter , you can confirm the manual of the bwa mem
  
  > **Comments:**
  > **username_1**: This isn't true. Samtools sort is perfectly capable of sorting from a pipe. Try it, just run `samtools view -h foo.bam | samtools sort -o bar.bam`. And it can do this with humongous files, it doesn't store in memory, it creates temporary files while sorting precisely to avoid the problem you mention.  
  > **username_3**: Sorry about that,actually i have tried many times in my machine, but that's not hpc,it just has 64GB memory, and I indeed got that problem,when i use the bwa mem | samtools view | samtools sort my computer will be down, i will can't connect it by ssh, and after long time ,it will recover,but that is uncontrollable ,and i hava find out some oom events during the analysis when i can connect it again. I have resolved it by the way i hava said.But there is an other question in his commandline which has been mentioned in the second paragraph in my answer last post.It seems that you mention it too.  
  > **username_1**: That can be caused by various things, but I can assure you, I have been running `bwa mem | samtools sort` in a professional, production pipeline for almost a decade now. And also did so on my laptop, before posting, just to double check. I also confirmed in the [samtools manual\(https://www.htslib.org/doc/samtools-sort.html) that it creates temp files when memory is not enough. As for the last paragraph, I'm sorry but I don't understand it at all. Yes, the `-r` isn't needed, but it isn't causing the problem either.  
  > **username_3**: Thanks to reply, actually i have noted the directory which he should not use between the reference and the reads file too. But the words may be unclear.  On the other hand i major in computer science ,when i get the  problem i have mentioned and take account for the behaviour of my comuputer and the syslog, i think i could be the reason what i have said , and by split the command line i resolve it. So it makes me believe the reason more.  The version of the samtools and htslib is 1.20. I will try to do reproduction experiments to located the ture reason. Thank you for reminding me.  
  
  ---
  
  