Answers.Body: <p>Using the TblToFasta and FastaToTbl scripts <a href="https://bioinformatics.stackexchange.com/a/2651/298">I have posted before</a>, combined with classic *nix utils, you can do:</p>
  
  <pre><code>$ join -a1 -a2 &lt;(FastaToTbl dna1.fa | sort) &lt;(FastaToTbl dna2.fa | sort) | 
      sed 's/ /=/2' | TblToFasta | tr = '\n'
  &gt;chr1 
  ACTGACTAGCTAGCTAACTG
  GCATCGTAGCTAGCTACGAT
  &gt;chr2 
  GCATCGTAGCTAGCTACGAT
  &gt;chr3 
  CATCGATCGTACGTACGTAG
  CATCGATCGTACGTACGTAG
  &gt;chr4 
  ATCGATCGATCGTACGATCG
  ATCGATCGATCGTACGATCG
  &gt;chr5 
  CATCGATCGTACGTACGTAG
  </code></pre>
  
  <h3>Explanation</h3>
  
  <p>The <code>FastaToTbl</code> will put everything on the same line, with the sequence's ID at the beginning of the line, then a tab and then the sequence (no matter how many lines of sequence you have). So, we do this to both files, sort them and pass that as input to <code>join</code>, which will join its input on lines where the 1st field is the same (by default). The <code>-a1 -a2</code> means "also print non-matching lines from both files" so we don't miss cases where a sequence is in only one of the two files. </p>
  
  <p>Since the <code>join</code> will add a space between the two joined lines, we deal with that by replacing the second space on each line with a <code>=</code> (<code>sed 's/ /=/2'</code>) and then pass the whole thing to <code>TblToFasta</code> to revert to fasta format. Finally, we use <code>tr</code> to replace the <code>=</code> with a newline so you can see where each sequence came from as you show in your example.</p>
  
  <p>For longer sequences where you want to avoid sorting the input, you can do:</p>
  
  <pre><code>$ FastaToTbl dna1.fa dna2.fa | 
      awk '{
              seq[$1] ? seq[$1]=seq[$1]"\n"$2 : seq[$1]=$2 
           }
           END{
                for(chr in seq){
                  printf "&gt;%s\n%s\n", chr, seq[chr]
                }
           }'
  &gt;chr1
  ACTGACTAGCTAGCTAACTG
  GCATCGTAGCTAGCTACGAT
  &gt;chr2
  GCATCGTAGCTAGCTACGAT
  &gt;chr3
  CATCGATCGTACGTACGTAG
  CATCGATCGTACGTACGTAG
  &gt;chr4
  ATCGATCGATCGTACGATCG
  ATCGATCGATCGTACGATCG
  &gt;chr5
  CATCGATCGTACGTACGTAG
  </code></pre>
  
  <h3>Explanation</h3>
  
  <p>Here, the tbl format sequences (see above) are passed to an awk script which does the work:</p>
  
  <ul>
  <li><code>seq[$1] ? seq[$1]=seq[$1]"\n"$2 : seq[$1]=$2</code> : if the 1st field (<code>$1</code>, the sequence identifier) is already in the array <code>seq</code>, then set the value of <code>seq[$1]</code> to the current value (so whatever sequence has been already seen for this ID), a newline and the 2nd field (<code>$2</code>, the sequence). If it hasn't been seen before, then set the value of <code>seq[$1]</code> to the 2nd field, the sequence. </li>
  <li><code>END{ for(chr in seq){ printf "&gt;%s\n%s\n", chr, seq[chr]}</code> : after processing everything, go through each element in the array <code>seq</code> and print out the data in fasta format.</li>
  </ul>
  
  <p>However, that will mean keeping both files in memory which may be a problem for huge sequences. You can avoid that by using temp files:</p>
  
  <pre><code>$ FastaToTbl dna1.fa dna2.fa | 
      awk '{ 
              if(!seen[$1]++){
                  print "&gt;"$1 &gt; $1".tmp"
              } 
              print $2 &gt;&gt; $1".tmp"
          }' &amp;&amp; cat *.tmp &gt; out.fa &amp;&amp; rm *.tmp
  $ cat out.fa 
  &gt;chr1
  ACTGACTAGCTAGCTAACTG
  GCATCGTAGCTAGCTACGAT
  &gt;chr2
  GCATCGTAGCTAGCTACGAT
  &gt;chr3
  CATCGATCGTACGTACGTAG
  CATCGATCGTACGTACGTAG
  &gt;chr4
  ATCGATCGATCGTACGATCG
  ATCGATCGATCGTACGATCG
  &gt;chr5
  CATCGATCGTACGTACGTAG
  </code></pre>
  
  <h3>Explanation</h3>
  
  <p>This is the same basic idea as the script above, only instead of storing the information in an array, we print it as it comes into temp files. </p>
  
  <ul>
  <li><code>if(!seen[$1]++){print "&gt;"$1 &gt; $1".tmp"}</code> : if this is the first time we've seen this ID, print it as a fasta header into the file <code>$1".tmp"</code> (e.g. <code>chr1.tmp</code>).</li>
  <li><code>print $2 &gt;&gt; $1".tmp"</code> append the current sequence to the corresponding temp file.</li>
  </ul>
  
  <p>Once the temp files have been created, we concatenate them (<code>cat *.tmp &gt; out.fa</code>) and delete them (<code>rm *tmp</code>).</p>
  
  <hr>
  
  <p>As for why you would want to do this, an obvious example is reconstructing a reference genome from short sequences. You would want all the sequences of the same schromosome together. Of course, you would also need to assemble them in the right order, which you aren't doing here.</p>
  
  <p>Alternatively, you might want to concatenate them this way so as to avoid having the extra, unneeded lines (<code>&gt;chrN</code>) which would make your file larger. Or you would want to be able to retrieve all sequences with the same ID at once. </p>
  
  <p>But I don't really know why you would want to do it. You're the one asking for it, after all! :)</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-06-27T14:10:52.973Z
Answers.Text: Can you answer their question 2, the why?
Answers.UserDisplayName: Chris_Rands
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-06-27T11:08:11.157Z
Answers.LastActivityDate: 2018-07-02T07:42:38.807Z
Answers.LastEditDate: 2018-07-02T07:42:38.807Z
Answers.UserDisplayName: terdon
Body: <p>I have the following two small FASTAs, <code>dna1.fa</code> and <code>dna2.fa</code>:</p>
  
  <pre><code># dna1.fa
  &gt;chr1
  ACTGACTAGCTAGCTAACTG
  &gt;chr2
  GCATCGTAGCTAGCTACGAT
  &gt;chr3
  CATCGATCGTACGTACGTAG
  &gt;chr4
  ATCGATCGATCGTACGATCG
  
  # dna2.fa
  &gt;chr1
  GCATCGTAGCTAGCTACGAT
  &gt;chr3
  CATCGATCGTACGTACGTAG
  &gt;chr4
  ATCGATCGATCGTACGATCG
  &gt;chr5
  CATCGATCGTACGTACGTAG
  </code></pre>
  
  <p>If I concatenate these two files using <code>cat</code>, the piped result is as follows:</p>
  
  <pre><code>$ cat dna1.fa dna2.fa &gt; total1.fa
  $ cat total1.fa
  &gt;chr1
  ACTGACTAGCTAGCTAACTG
  &gt;chr2
  GCATCGTAGCTAGCTACGAT
  &gt;chr3
  CATCGATCGTACGTACGTAG
  &gt;chr4
  ATCGATCGATCGTACGATCG
  &gt;chr1
  GCATCGTAGCTAGCTACGAT
  &gt;chr3
  CATCGATCGTACGTACGTAG
  &gt;chr4
  ATCGATCGATCGTACGATCG
  &gt;chr5
  CATCGATCGTACGTACGTAG
  </code></pre>
  
  <p>Here, as expected, these are two files which have concatenated together. It seems this is usually what is done when one concatenates several FASTAs together in bioinformatics.</p>
  
  <p>Question1: Is it possible to concatenate these "based on the chromosome"? Here is the desired result:</p>
  
  <pre><code>&gt;chr1
  ACTGACTAGCTAGCTAACTG
  GCATCGTAGCTAGCTACGAT
  &gt;chr2
  GCATCGTAGCTAGCTACGAT
  &gt;chr3
  CATCGATCGTACGTACGTAG
  CATCGATCGTACGTACGTAG
  &gt;chr4
  ATCGATCGATCGTACGATCG
  ATCGATCGATCGTACGATCG
  &gt;chr5
  CATCGATCGTACGTACGTAG
  </code></pre>
  
  <p>Note that these are exceptionally small FASTAs---usually one could be dealing with files of several GB in size. </p>
  
  <p>Question2: The meta question here is, are there any standard bioinformatic analyses whereby the "concatenated by chromosome" FASTA would be more useful than the straightforward concatenated FASTA via <code>cat</code> alone? </p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-06-25T14:28:20.473Z
Comments.Text: Why would you do this? You're creating contiguous sequences out of (presumably) non-contiguous sequence?
Comments.UserDisplayName: Chris_Rands
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-06-25T14:28:55.290Z
Comments.Text: Also you should not assume FASTA format is one sequence per line
Comments.UserDisplayName: Chris_Rands
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-06-25T14:29:46.563Z
Comments.Text: What's the actual context here? Are these sequence alignments? Aligned FASTA would not follow the format of your desired output
Comments.UserDisplayName: Chris_Rands
Tags: <fasta>
Title: How to concatenate FASTAs with respect to chromosome?
ThreadText: # How to concatenate FASTAs with respect to chromosome?
  
  (Asked by: username_0 on 2018-06-25)
  
  I have the following two small FASTAs, `dna1.fa` and `dna2.fa`:
  
  ```
  # dna1.fa
  >chr1
  ACTGACTAGCTAGCTAACTG
  >chr2
  GCATCGTAGCTAGCTACGAT
  >chr3
  CATCGATCGTACGTACGTAG
  >chr4
  ATCGATCGATCGTACGATCG
  
  # dna2.fa
  >chr1
  GCATCGTAGCTAGCTACGAT
  >chr3
  CATCGATCGTACGTACGTAG
  >chr4
  ATCGATCGATCGTACGATCG
  >chr5
  CATCGATCGTACGTACGTAG
  
  ```
  
  If I concatenate these two files using `cat`, the piped result is as follows:
  
  ```
  $ cat dna1.fa dna2.fa > total1.fa
  $ cat total1.fa
  >chr1
  ACTGACTAGCTAGCTAACTG
  >chr2
  GCATCGTAGCTAGCTACGAT
  >chr3
  CATCGATCGTACGTACGTAG
  >chr4
  ATCGATCGATCGTACGATCG
  >chr1
  GCATCGTAGCTAGCTACGAT
  >chr3
  CATCGATCGTACGTACGTAG
  >chr4
  ATCGATCGATCGTACGATCG
  >chr5
  CATCGATCGTACGTACGTAG
  
  ```
  
  Here, as expected, these are two files which have concatenated together. It seems this is usually what is done when one concatenates several FASTAs together in bioinformatics.
  
  Question1: Is it possible to concatenate these "based on the chromosome"? Here is the desired result:
  
  ```
  >chr1
  ACTGACTAGCTAGCTAACTG
  GCATCGTAGCTAGCTACGAT
  >chr2
  GCATCGTAGCTAGCTACGAT
  >chr3
  CATCGATCGTACGTACGTAG
  CATCGATCGTACGTACGTAG
  >chr4
  ATCGATCGATCGTACGATCG
  ATCGATCGATCGTACGATCG
  >chr5
  CATCGATCGTACGTACGTAG
  
  ```
  
  Note that these are exceptionally small FASTAs---usually one could be dealing with files of several GB in size.
  
  Question2: The meta question here is, are there any standard bioinformatic analyses whereby the "concatenated by chromosome" FASTA would be more useful than the straightforward concatenated FASTA via `cat` alone?
  
  **Tags:** fasta
  
  > **Comments:**
  > **username_1**: Why would you do this? You're creating contiguous sequences out of (presumably) non-contiguous sequence?  
  > **username_1**: Also you should not assume FASTA format is one sequence per line  
  > **username_1**: What's the actual context here? Are these sequence alignments? Aligned FASTA would not follow the format of your desired output  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2018-06-27. Score: 2 | Accepted answer
  
  Using the TblToFasta and FastaToTbl scripts [I have posted before](https://bioinformatics.stackexchange.com/a/2651/298), combined with classic \*nix utils, you can do:
  
  ```
  $ join -a1 -a2 <(FastaToTbl dna1.fa | sort) <(FastaToTbl dna2.fa | sort) | 
      sed 's/ /=/2' | TblToFasta | tr = '\n'
  >chr1 
  ACTGACTAGCTAGCTAACTG
  GCATCGTAGCTAGCTACGAT
  >chr2 
  GCATCGTAGCTAGCTACGAT
  >chr3 
  CATCGATCGTACGTACGTAG
  CATCGATCGTACGTACGTAG
  >chr4 
  ATCGATCGATCGTACGATCG
  ATCGATCGATCGTACGATCG
  >chr5 
  CATCGATCGTACGTACGTAG
  
  ```
  
  ### Explanation
  
  The `FastaToTbl` will put everything on the same line, with the sequence's ID at the beginning of the line, then a tab and then the sequence (no matter how many lines of sequence you have). So, we do this to both files, sort them and pass that as input to `join`, which will join its input on lines where the 1st field is the same (by default). The `-a1 -a2` means "also print non-matching lines from both files" so we don't miss cases where a sequence is in only one of the two files.
  
  Since the `join` will add a space between the two joined lines, we deal with that by replacing the second space on each line with a `=` (`sed 's/ /=/2'`) and then pass the whole thing to `TblToFasta` to revert to fasta format. Finally, we use `tr` to replace the `=` with a newline so you can see where each sequence came from as you show in your example.
  
  For longer sequences where you want to avoid sorting the input, you can do:
  
  ```
  $ FastaToTbl dna1.fa dna2.fa | 
      awk '{
              seq[$1] ? seq[$1]=seq[$1]"\n"$2 : seq[$1]=$2 
           }
           END{
                for(chr in seq){
                  printf ">%s\n%s\n", chr, seq[chr]
                }
           }'
  >chr1
  ACTGACTAGCTAGCTAACTG
  GCATCGTAGCTAGCTACGAT
  >chr2
  GCATCGTAGCTAGCTACGAT
  >chr3
  CATCGATCGTACGTACGTAG
  CATCGATCGTACGTACGTAG
  >chr4
  ATCGATCGATCGTACGATCG
  ATCGATCGATCGTACGATCG
  >chr5
  CATCGATCGTACGTACGTAG
  
  ```
  
  ### Explanation
  
  Here, the tbl format sequences (see above) are passed to an awk script which does the work:
  
  * `seq[$1] ? seq[$1]=seq[$1]"\n"$2 : seq[$1]=$2` : if the 1st field (`$1`, the sequence identifier) is already in the array `seq`, then set the value of `seq[$1]` to the current value (so whatever sequence has been already seen for this ID), a newline and the 2nd field (`$2`, the sequence). If it hasn't been seen before, then set the value of `seq[$1]` to the 2nd field, the sequence.
  * `END{ for(chr in seq){ printf ">%s\n%s\n", chr, seq[chr]}` : after processing everything, go through each element in the array `seq` and print out the data in fasta format.
  
  However, that will mean keeping both files in memory which may be a problem for huge sequences. You can avoid that by using temp files:
  
  ```
  $ FastaToTbl dna1.fa dna2.fa | 
      awk '{ 
              if(!seen[$1]++){
                  print ">"$1 > $1".tmp"
              } 
              print $2 >> $1".tmp"
          }' && cat *.tmp > out.fa && rm *.tmp
  $ cat out.fa 
  >chr1
  ACTGACTAGCTAGCTAACTG
  GCATCGTAGCTAGCTACGAT
  >chr2
  GCATCGTAGCTAGCTACGAT
  >chr3
  CATCGATCGTACGTACGTAG
  CATCGATCGTACGTACGTAG
  >chr4
  ATCGATCGATCGTACGATCG
  ATCGATCGATCGTACGATCG
  >chr5
  CATCGATCGTACGTACGTAG
  
  ```
  
  ### Explanation
  
  This is the same basic idea as the script above, only instead of storing the information in an array, we print it as it comes into temp files.
  
  * `if(!seen[$1]++){print ">"$1 > $1".tmp"}` : if this is the first time we've seen this ID, print it as a fasta header into the file `$1".tmp"` (e.g. `chr1.tmp`).
  * `print $2 >> $1".tmp"` append the current sequence to the corresponding temp file.
  
  Once the temp files have been created, we concatenate them (`cat *.tmp > out.fa`) and delete them (`rm *tmp`).
  
  ---
  
  As for why you would want to do this, an obvious example is reconstructing a reference genome from short sequences. You would want all the sequences of the same schromosome together. Of course, you would also need to assemble them in the right order, which you aren't doing here.
  
  Alternatively, you might want to concatenate them this way so as to avoid having the extra, unneeded lines (`>chrN`) which would make your file larger. Or you would want to be able to retrieve all sequences with the same ID at once.
  
  But I don't really know why you would want to do it. You're the one asking for it, after all! :)
  
  > **Comments:**
  > **username_1**: Can you answer their question 2, the why?  
  
  ---
  
  