Answers.Body: <p><strong>Not so elegant but working solution</strong></p>
  
  <p>I found a solution that satisfy several of my conditions, basically I just have to assign read groups to individual mapping files, which can be just added to sorting process and then merge the sorted sam files with read groups.</p>
  
  <pre class="lang-js prettyprint-override"><code>function sort_and_assign_RG {
      # $1 input file
      # $2 read group id
      # $3 library
  
      OFILE=$(basename $1 .sam).sort.RG.sam
      HEADER=$(basename $1 .sam).header.sam
  
      # since I take header from unsorted file, so I need to add this information to header as well as the readgroup
      echo -e "@HD\tVN:1.3\tSO:coordinate" &gt; $HEADER
      # add read group to header
      echo -e "@RG\tID:"$2"\tLB:""$3" &gt;&gt; $HEADER
      # and the rest of the header
      samtools view -H $1 &gt;&gt; $HEADER
  
      # now sort the input sam, remove header, 
      # attach to each alignment read group ID (RGID) 
      # and cat header and alignment together
      samtools sort -- $1 | samtools view - | \
        awk -v RGID="$2" '{ printf "%s\tRG:Z:%s\n", $0, RGID; }' | \
        cat $HEADER - &gt; $OFILE
  
      # remove temp header
      rm $HEADER
  }
  
  sort_and_assign_RG s1.sam s1 is180
  sort_and_assign_RG s2.sam s2 is350
  sort_and_assign_RG s3.sam s3 is550
  
  samtools merge merged.bam s{1,2,3}.sort.RG.sam
  </code></pre>
  
  <p>It would be nice to avoid creation of three <code>sam</code> files on the way, but at least I get standardized read groups in a merged bam tile.</p>
  
  <p><em>--- edit ---</em></p>
  
  <p>I also found where my confusion is coming from. The read group in header is in format </p>
  
  <pre><code>@RG    ID:foo    LB:lib_foo    ...`
  @RG    ID:bar    LB:lib_bar    ...`
  </code></pre>
  
  <p>where <code>ID</code> is read group ID (not sample ID as I thought) and other tags are just specifications of the read group. Then individual mapping have read group assigned in different format</p>
  
  <pre><code>{aliment1}    RG:Z:foo
  {aliment2}    RG:Z:bar
  </code></pre>
  
  <p>where <code>RG</code> is just a tag for read group and <code>Z</code> is just a mark that says that this tag is just a "printable string". Therefore I think that another solution would just to merge these sorted files and then just adding three <strong>correct</strong> lines to the header.</p>
  
  <p>I have to say, <code>.sam</code> is a unbelievable intuitive format.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-06-27T11:11:42.990Z
Answers.Text: You may try to use named pipes instead of files for your sam files: `mkfifo $OFILE` before the `samtools sort`. Then you call your `sort_and_assign_RG` in the background with `&` and do `rm -f s{1,2,3}.sort.RG.sam` after the `samtools merge`. I'm not sure this will really work in parallel, though: I suspect that the named pipes can be written only once the `samtools sort` are completed.
Answers.UserDisplayName: bli
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-06-26T16:50:19.213Z
Answers.LastActivityDate: 2017-06-26T17:23:21.913Z
Answers.LastEditDate: 2017-06-26T17:23:21.913Z
Answers.UserDisplayName: Kamil S Jaron
Answers.Body: <p>If the files already have defined read groups inside them, then doing a merge without the <code>-r</code> option should work:</p>
  <pre><code>samtools merge merged.bam s1.sort.sam s2.sort.sam s3.sort.sam
  </code></pre>
  <p>If there are multiple input files that share the same read group, then by default they will have random strings appended to make the read groups unique. This can be stopped by using the <code>-c</code> option, as mentioned in <code>man samtools merge</code>:</p>
  <pre><code>   -c      When  several  input files contain @RG headers with the same ID, emit only
             one of them (namely, the header line from the first file we find  that  ID
             in) to the merged output file.  Combining these similar headers is usually
             the right thing to do when the files being merged originated from the same
             file.
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-08-20T06:38:35.903Z
Answers.LastActivityDate: 2022-08-20T06:38:35.903Z
Answers.UserDisplayName: gringer
Answers.Body: <p>I have used <a href="https://github.com/ekg/bamaddrg" rel="noreferrer">https://github.com/ekg/bamaddrg</a> to add read groups quickly to multiple sam files.  And then you can do a samtools merge of the tagged files.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-06-27T08:47:44.440Z
Answers.Text: This looks like elegant and faster version of `awk`. I have to take a look!
Answers.UserDisplayName: Kamil S Jaron
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-06-27T04:10:47.523Z
Answers.LastActivityDate: 2017-06-27T04:10:47.523Z
Answers.UserDisplayName: Bioathlete
Body: <p>I have three sequencing libraries of single individual mapped to a reference using <code>bwa-mem</code>. I would like to merge the three unsorted <code>.sam</code> files I have so, I can call variants and heterozygosity estimates using <a href="https://bitbucket.org/phaentu/atlas/wiki/Home" rel="nofollow noreferrer">atlas</a>. Atlas requires one input mapping file (<code>bam</code>) with defined read groups because it estimates the error profiles of different libraries separately.</p>
  
  <p>How can I merge multiple sam files? Preferably avoiding java (Picard tools).</p>
  
  <p>I tried to figure out a solution using <code>samtools 1.3</code>. I sorted individual files using <code>samtools sort</code>, then I used <code>samtools merge -r merged.bam s1.sort.sam s2.sort.sam s3.sort.sam</code> to merge the sorted files. However, the read group didn't make it to the header (and the variant caller I use is complaining about it), also the read group is stupidly the file name.</p>
  
  <p>I tried to define meaningful readgroup names using procedure described on <a href="https://www.biostars.org/p/47487/" rel="nofollow noreferrer">BioStars</a>, but I found that this will just change the header, it does not adjust the names of read groups defined by <code>samtools merge</code> (the file names).</p>
  
  <p>Following this related thread on <a href="http://seqanswers.com/forums/showthread.php?t=4180" rel="nofollow noreferrer">SeqAnswers</a>, I tried to define the correct header with read groups corresponding to merged file names:</p>
  
  <pre><code>samtools -rh rg.txt merged.bam s1.sort.sam s2.sort.sam s3.sort.sam
  </code></pre>
  
  <p>where <code>rg.txt</code> is a file with header</p>
  
  <pre><code>@RG s1.sort
  @RG s2.sort
  @RG s3.sort
  ...
  output of samtools view -H s1.sort
  </code></pre>
  
  <p>However, the header still had not the read group, I guess because sam header accepts only tagged items to be specified (something like <code>@RG    XY:s1.sort</code>). So, I looked into the merged <code>bam</code> and I found that the tag of RG is <code>Z:</code>. So I tried to just rename header of the merged file using <code>samtools reheader</code>, but then samtools complain about the fact that a tag needs to be of length 2:</p>
  
  <pre><code>Malformed key:value pair at line 123: "@RG    Z:s1.sort"
  Segmentation fault (core dumped)
  </code></pre>
  
  <p>I have opened an <a href="https://github.com/samtools/samtools/issues/698" rel="nofollow noreferrer">issue</a> to report this strange incompatibility of read groups generated by <code>samtools merge</code> with <code>samtools reheader</code>.</p>
  
  <p>I would like solution to :</p>
  
  <ul>
  <li>create a bit more standardized read group names (<code>SM:Sample\tLB:library</code> format)</li>
  <li>avoid pointless writing to disk like in sam -> sorted.sam -> merged.bam case (can be probably achieved through "pipes and tees", thanks @bli)</li>
  </ul>
  
  <p>I also know that I can specify RG to bwa, so the sam files will have read groups defined in the first place. But I do not like the idea of remapping three libraries just to create correct formatting of read groups.</p>
  
Tags: <sam>
Title: How to merge sam files together with adding read groups
ThreadText: # How to merge sam files together with adding read groups
  
  (Asked by: username_0 on 2017-06-26)
  
  I have three sequencing libraries of single individual mapped to a reference using `bwa-mem`. I would like to merge the three unsorted `.sam` files I have so, I can call variants and heterozygosity estimates using [atlas](https://bitbucket.org/phaentu/atlas/wiki/Home). Atlas requires one input mapping file (`bam`) with defined read groups because it estimates the error profiles of different libraries separately.
  
  How can I merge multiple sam files? Preferably avoiding java (Picard tools).
  
  I tried to figure out a solution using `samtools 1.3`. I sorted individual files using `samtools sort`, then I used `samtools merge -r merged.bam s1.sort.sam s2.sort.sam s3.sort.sam` to merge the sorted files. However, the read group didn't make it to the header (and the variant caller I use is complaining about it), also the read group is stupidly the file name.
  
  I tried to define meaningful readgroup names using procedure described on [BioStars](https://www.biostars.org/p/47487/), but I found that this will just change the header, it does not adjust the names of read groups defined by `samtools merge` (the file names).
  
  Following this related thread on [SeqAnswers](http://seqanswers.com/forums/showthread.php?t=4180), I tried to define the correct header with read groups corresponding to merged file names:
  
  ```
  samtools -rh rg.txt merged.bam s1.sort.sam s2.sort.sam s3.sort.sam
  
  ```
  
  where `rg.txt` is a file with header
  
  ```
  @RG s1.sort
  @RG s2.sort
  @RG s3.sort
  ...
  output of samtools view -H s1.sort
  
  ```
  
  However, the header still had not the read group, I guess because sam header accepts only tagged items to be specified (something like `@RG XY:s1.sort`). So, I looked into the merged `bam` and I found that the tag of RG is `Z:`. So I tried to just rename header of the merged file using `samtools reheader`, but then samtools complain about the fact that a tag needs to be of length 2:
  
  ```
  Malformed key:value pair at line 123: "@RG    Z:s1.sort"
  Segmentation fault (core dumped)
  
  ```
  
  I have opened an [issue](https://github.com/samtools/samtools/issues/698) to report this strange incompatibility of read groups generated by `samtools merge` with `samtools reheader`.
  
  I would like solution to :
  
  * create a bit more standardized read group names (`SM:Sample\tLB:library` format)
  * avoid pointless writing to disk like in sam -> sorted.sam -> merged.bam case (can be probably achieved through "pipes and tees", thanks @username_1)
  
  I also know that I can specify RG to bwa, so the sam files will have read groups defined in the first place. But I do not like the idea of remapping three libraries just to create correct formatting of read groups.
  
  **Tags:** sam
  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_3 on 2017-06-27. Score: 5 | Accepted answer
  
  I have used <https://github.com/ekg/bamaddrg> to add read groups quickly to multiple sam files. And then you can do a samtools merge of the tagged files.
  
  > **Comments:**
  > **username_0**: This looks like elegant and faster version of `awk`. I have to take a look!  
  
  ---
  
  ### Answer by username_0 on 2017-06-26. Score: 3
  
  **Not so elegant but working solution**
  
  I found a solution that satisfy several of my conditions, basically I just have to assign read groups to individual mapping files, which can be just added to sorting process and then merge the sorted sam files with read groups.
  
  ```js
  function sort_and_assign_RG {
      # $1 input file
      # $2 read group id
      # $3 library
  
      OFILE=$(basename $1 .sam).sort.RG.sam
      HEADER=$(basename $1 .sam).header.sam
  
      # since I take header from unsorted file, so I need to add this information to header as well as the readgroup
      echo -e "@HD\tVN:1.3\tSO:coordinate" > $HEADER
      # add read group to header
      echo -e "@RG\tID:"$2"\tLB:""$3" >> $HEADER
      # and the rest of the header
      samtools view -H $1 >> $HEADER
  
      # now sort the input sam, remove header, 
      # attach to each alignment read group ID (RGID) 
      # and cat header and alignment together
      samtools sort -- $1 | samtools view - | \
        awk -v RGID="$2" '{ printf "%s\tRG:Z:%s\n", $0, RGID; }' | \
        cat $HEADER - > $OFILE
  
      # remove temp header
      rm $HEADER
  }
  
  sort_and_assign_RG s1.sam s1 is180
  sort_and_assign_RG s2.sam s2 is350
  sort_and_assign_RG s3.sam s3 is550
  
  samtools merge merged.bam s{1,2,3}.sort.RG.sam
  
  ```
  
  It would be nice to avoid creation of three `sam` files on the way, but at least I get standardized read groups in a merged bam tile.
  
  *--- edit ---*
  
  I also found where my confusion is coming from. The read group in header is in format
  
  ```
  @RG    ID:foo    LB:lib_foo    ...`
  @RG    ID:bar    LB:lib_bar    ...`
  
  ```
  
  where `ID` is read group ID (not sample ID as I thought) and other tags are just specifications of the read group. Then individual mapping have read group assigned in different format
  
  ```
  {aliment1}    RG:Z:foo
  {aliment2}    RG:Z:bar
  
  ```
  
  where `RG` is just a tag for read group and `Z` is just a mark that says that this tag is just a "printable string". Therefore I think that another solution would just to merge these sorted files and then just adding three **correct** lines to the header.
  
  I have to say, `.sam` is a unbelievable intuitive format.
  
  > **Comments:**
  > **username_1**: You may try to use named pipes instead of files for your sam files: `mkfifo $OFILE` before the `samtools sort`. Then you call your `sort_and_assign_RG` in the background with `&` and do `rm -f s{1,2,3}.sort.RG.sam` after the `samtools merge`. I'm not sure this will really work in parallel, though: I suspect that the named pipes can be written only once the `samtools sort` are completed.  
  
  ---
  
  ### Answer by username_2 on 2022-08-20. Score: 0
  
  If the files already have defined read groups inside them, then doing a merge without the `-r` option should work:
  
  ```
  samtools merge merged.bam s1.sort.sam s2.sort.sam s3.sort.sam
  
  ```
  
  If there are multiple input files that share the same read group, then by default they will have random strings appended to make the read groups unique. This can be stopped by using the `-c` option, as mentioned in `man samtools merge`:
  
  ```
     -c      When  several  input files contain @RG headers with the same ID, emit only
             one of them (namely, the header line from the first file we find  that  ID
             in) to the merged output file.  Combining these similar headers is usually
             the right thing to do when the files being merged originated from the same
             file.
  
  ```
  
  
  ---
  
  