Body: <p>I am trying to do adapter trimming, alignment and sorting for a range of large scale  paired end fastq files. The code I am using is given below:</p>
  <pre><code>#!/bin/bash
  # Do Trimming, Alignment according to Config file
  
  #SBATCH --job-name=T_Align_single
  #SBATCH --nodes=1
  #SBATCH --ntasks=1
  #SBATCH --cpus-per-task=32
  #SBATCH --partition=batch
  #SBATCH --account=markay
  #SBATCH --time=72:00:00
  #SBATCH --mem=16G
  #SBATCH --mail-type=TIME_LIMIT_80,BEGIN,END
  #SBATCH --mail-user=&lt;email-redacted&gt;
  
  module load bowtie2/2.4.1 
  module load samtools/1.9
  
  mkdir -p Mapping/
  
  config=${INPUT_FILE}
  
  file1=<span class="math-container">$(cat $</span>config | awk '<span class="math-container">$1 ~ /fastq/ {print $1}')
  file2=$</span>(cat <span class="math-container">$config | awk '$</span>1 ~ /fastq/ {print <span class="math-container">$2}')
  file3=$</span>(cat <span class="math-container">$config | awk '$</span>1 ~ /index=/ {print $1}'| sed 's/ref_index=//')
  
  
  #adapter trimming
  trimfile1=Mapping/&quot;<span class="math-container">${file1%.fastq}".trim.fastq
  trimfile2=Mapping/"$</span>{file2%.fastq}&quot;.trim.fastq
  
   java -jar /labs/markay/Aranyak/Adriana_intro_CnT/Cnt_Test/Test-analysis/fastq_processing/Trimmomatic-0.39/trimmomatic-0.39.jar  PE -phred33 <span class="math-container">$file1 $</span>file2 \
     <span class="math-container">$trimfile1 Mapping/"$</span>{file1%.fastq}&quot;.unpaired.fastq \
     <span class="math-container">$trimfile2 Mapping/"$</span>{file2%.fastq}&quot;.unpaired.fastq \
     LEADING:3 TRAILING:3 SLIDINGWINDOW:4:15 \
     MINLEN:20 ILLUMINACLIP: /labs/markay/Aranyak/Adriana_intro_CnT/adapter.fa:2:30:10 -threads ${SLURM_CPUS_PER_TASK}
  
  # mapping using bowtie2
  sortbam=Mapping/&quot;<span class="math-container">${file1%_R1_001.fastq}".sort.bam
  sample_ID=$</span>(cat <span class="math-container">$config | awk '$</span>1 ~ /fastq/ {print <span class="math-container">$4}')
  bowtie2 -p $</span>{SLURM_CPUS_PER_TASK} --local --very-sensitive-local --no-unal \
     --no-mixed --no-discordant --phred33 -I 10 -X 700 \
     -x <span class="math-container">$file3 \
     -1 $</span>trimfile1 -2 <span class="math-container">$trimfile2 --rg-id $</span>sample_ID | samtools view -Sb - | samtools sort -@<span class="math-container">${SLURM_CPUS_PER_TASK} -o $</span>sortbam
  
  #Index the sorting by coordinate bam file
  samtools index ${sortbam}
  </code></pre>
  <p>My original fastq files before adapter trimming looked like this</p>
  <pre><code>R3-LiD-M-LK_S16_L002_R2_001.fastq.gz
  R3-LiD-M-LK_S16_L002_R1_001.fastq.gz
  </code></pre>
  <p>However after adapter trimming I removed the lane names and my file looks like this</p>
  <pre><code>R3-LiD-M-LK_S16_R2_001.fastq.gz
  R3-LiD-M-LK_S16_R1_001.fastq.gz
  </code></pre>
  <p>The code that worked for the original file is not generating a proper bam file at the end though I am getting a list of trimmed fastq files. The error message I am getting is:</p>
  <pre><code>[E::hts_open_format] Failed to open file R1-LiD-H-DJ_S3.sort.rmdup.bam
  samtools index: failed to open &quot;R1-LiD-H-DJ_S3.sort.rmdup.bam&quot;: No such file or directory
  [E::hts_open_format] Failed to open file R1-LiD-H-DJ_S3.sort.bam
  The file 'R1-LiD-H-DJ_S3.sort.bam' does not exist
  [E::hts_open_format] Failed to open file R1-LiD-H-DJ_S3.sort.rmdup.bam
  The file 'R1-LiD-H-DJ_S3.sort.rmdup.bam' does not exist
  R1-LiD-H-DJ_S3 DONE.
  </code></pre>
  <p>How should I change the code to generate proper alignment and bam files?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-09-19T22:00:45.700Z
Comments.Text: This should be a simple learning exercise for you - all that's changed is that the lane number part is gone, but your error doesn't seem related to that at all, and no variables use the lane number part anywhere. Add a `set -x` after the `#SBATCH` lines and look at the job's STDERR file to see the exact commands being executed.
Comments.UserDisplayName: Ram RS
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-09-21T16:40:16.887Z
Comments.Text: Or just print out your variables so you can see what is happening. You are trying to open a file named `R1-LiD-H-DJ_S3.sort.bam` and a file named `R1-LiD-H-DJ_S3.sort.rmdup.bam` and neither of them exist. You also don't mention `rmdub` anywhere in your code, is this _really_ the code you are running?
Comments.UserDisplayName: terdon
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-09-21T23:54:51.430Z
Comments.Text: Thanks for your help, I have been able to trouble shoot the error with the naming of the files.
Comments.UserDisplayName: Aranyak Goswami
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-10-13T23:38:37.407Z
Comments.Text: Hi @AranyakGoswami, can you please write up your solution as an answer to this question (and make it an accepted answer)? That will help the StackExchange bots do the right thing with regards to not hunting for more answers to an already-solved question.
Comments.UserDisplayName: gringer
Tags: <ngs><genomics><rna>
Title: Adapter trimming
ThreadText: # Adapter trimming
  
  (Asked by: username_0 on 2022-09-18)
  
  I am trying to do adapter trimming, alignment and sorting for a range of large scale paired end fastq files. The code I am using is given below:
  
  ```
  #!/bin/bash
  # Do Trimming, Alignment according to Config file
  
  #SBATCH --job-name=T_Align_single
  #SBATCH --nodes=1
  #SBATCH --ntasks=1
  #SBATCH --cpus-per-task=32
  #SBATCH --partition=batch
  #SBATCH --account=markay
  #SBATCH --time=72:00:00
  #SBATCH --mem=16G
  #SBATCH --mail-type=TIME_LIMIT_80,BEGIN,END
  #SBATCH --mail-user=<email-redacted>
  
  module load bowtie2/2.4.1 
  module load samtools/1.9
  
  mkdir -p Mapping/
  
  config=${INPUT_FILE}
  
  file1=$(cat $config | awk '$1 ~ /fastq/ {print $1}')
  file2=$(cat $config | awk '$1 ~ /fastq/ {print $2}')
  file3=$(cat $config | awk '$1 ~ /index=/ {print $1}'| sed 's/ref_index=//')
  
  #adapter trimming
  trimfile1=Mapping/"${file1%.fastq}".trim.fastq
  trimfile2=Mapping/"${file2%.fastq}".trim.fastq
  
   java -jar /labs/markay/username_0/Adriana_intro_CnT/Cnt_Test/Test-analysis/fastq_processing/Trimmomatic-0.39/trimmomatic-0.39.jar  PE -phred33 $file1 $file2 \
     $trimfile1 Mapping/"${file1%.fastq}".unpaired.fastq \
     $trimfile2 Mapping/"${file2%.fastq}".unpaired.fastq \
     LEADING:3 TRAILING:3 SLIDINGWINDOW:4:15 \
     MINLEN:20 ILLUMINACLIP: /labs/markay/username_0/Adriana_intro_CnT/adapter.fa:2:30:10 -threads ${SLURM_CPUS_PER_TASK}
  
  # mapping using bowtie2
  sortbam=Mapping/"${file1%_R1_001.fastq}".sort.bam
  sample_ID=$(cat $config | awk '$1 ~ /fastq/ {print $4}')
  bowtie2 -p ${SLURM_CPUS_PER_TASK} --local --very-sensitive-local --no-unal \
     --no-mixed --no-discordant --phred33 -I 10 -X 700 \
     -x $file3 \
     -1 $trimfile1 -2 $trimfile2 --rg-id $sample_ID | samtools view -Sb - | samtools sort -@${SLURM_CPUS_PER_TASK} -o $sortbam
  
  #Index the sorting by coordinate bam file
  samtools index ${sortbam}
  
  ```
  
  My original fastq files before adapter trimming looked like this
  
  ```
  R3-LiD-M-LK_S16_L002_R2_001.fastq.gz
  R3-LiD-M-LK_S16_L002_R1_001.fastq.gz
  
  ```
  
  However after adapter trimming I removed the lane names and my file looks like this
  
  ```
  R3-LiD-M-LK_S16_R2_001.fastq.gz
  R3-LiD-M-LK_S16_R1_001.fastq.gz
  
  ```
  
  The code that worked for the original file is not generating a proper bam file at the end though I am getting a list of trimmed fastq files. The error message I am getting is:
  
  ```
  [E::hts_open_format] Failed to open file R1-LiD-H-DJ_S3.sort.rmdup.bam
  samtools index: failed to open "R1-LiD-H-DJ_S3.sort.rmdup.bam": No such file or directory
  [E::hts_open_format] Failed to open file R1-LiD-H-DJ_S3.sort.bam
  The file 'R1-LiD-H-DJ_S3.sort.bam' does not exist
  [E::hts_open_format] Failed to open file R1-LiD-H-DJ_S3.sort.rmdup.bam
  The file 'R1-LiD-H-DJ_S3.sort.rmdup.bam' does not exist
  R1-LiD-H-DJ_S3 DONE.
  
  ```
  
  How should I change the code to generate proper alignment and bam files?
  
  **Tags:** ngs, genomics, rna
  
  > **Comments:**
  > **username_1**: This should be a simple learning exercise for you - all that's changed is that the lane number part is gone, but your error doesn't seem related to that at all, and no variables use the lane number part anywhere. Add a `set -x` after the `#SBATCH` lines and look at the job's STDERR file to see the exact commands being executed.  
  > **username_2**: Or just print out your variables so you can see what is happening. You are trying to open a file named `R1-LiD-H-DJ_S3.sort.bam` and a file named `R1-LiD-H-DJ_S3.sort.rmdup.bam` and neither of them exist. You also don't mention `rmdub` anywhere in your code, is this _really_ the code you are running?  
  > **username_0**: Thanks for your help, I have been able to trouble shoot the error with the naming of the files.  
  > **username_3**: Hi @username_0, can you please write up your solution as an answer to this question (and make it an accepted answer)? That will help the StackExchange bots do the right thing with regards to not hunting for more answers to an already-solved question.  
  
  ---
  
  ## 0 answers
  
  