Answers.Body: <p>An unmapped SAM file format allows for read groups. You can use <code>samtools import</code> to create unmapped SAM files from FASTQ files:</p>
  <pre><code>samtools import -R virus-1-2 in12.fq &gt; 12_reads.sam
  samtools import -R virus-1-2-3 in123.fq &gt; 123_reads.sam
  samtools merge out.sam 12_reads.sam 123_reads.sam
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-08-20T12:28:07.313Z
Answers.Text: Perfect, thank you. It can be manipulated via Biopython `SamtoolsCalmdCommandline`
Answers.UserDisplayName: M__
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-08-20T06:24:19.657Z
Answers.LastActivityDate: 2022-08-20T06:24:19.657Z
Answers.UserDisplayName: gringer
Answers.Body: <p>I'm awarding 'solved' to @gringer, because thats the context of the question.</p>
  <p>The best solution I could find is stacking <code>Stockholm (sth)</code>  formats together. In this example the RNA families are associated with one structure:</p>
  <pre><code># STOCKHOLM 1.0
  #=GF ID    UPSK
  AF035635.1/619-641             UGAGUUCUCGAUCUCUAAAAUCG
  M24804.1/82-104                UGAGUUCUCUAUCUCUAAAAUCG
  J04373.1/6212-6234             UAAGUUCUCGAUCUUUAAAAUCG
  M24803.1/1-23                  UAAGUUCUCGAUCUCUAAAAUCG
  #=GC SS_cons                   .AAA....&lt;&lt;&lt;&lt;aaa....&gt;&gt;&gt;&gt;
  //
  </code></pre>
  <p><code>Biopython</code> allows <code>sth</code> format to be input.</p>
  <pre><code>from Bio import AlignIO
  from pathlib import Path
  
  file = 'test.sth'
  path = '/Volumes/mypath/testSth'
  alignment = AlignIO.read(open(Path(path, file)), &quot;stockholm&quot;)
  print(&quot;Alignment length %i&quot; % alignment.get_alignment_length())
  print (alignment.column_annotations['secondary_structure'])
  </code></pre>
  <p>Output,</p>
  <pre><code>Alignment length 23 
  .AAA....&lt;&lt;&lt;&lt;aaa....&gt;&gt;&gt;&gt; # this is the RNA structure 
  </code></pre>
  <p>If we look the <code>alignment</code> object via</p>
  <pre><code>from pprint import pprint
  pprint(vars(alignment)) # or via print(alignment.__dict__)
  </code></pre>
  <p>The output</p>
  <blockquote>
  <p>{'_annotations': {},  '<strong>_per_col_annotations</strong>': {<strong>'secondary_structure'</strong>:
  '.AAA....&lt;&lt;&lt;&lt;aaa....&gt;&gt;&gt;&gt;'},<br />
  '_records':
  [SeqRecord(seq=Seq('UGAGUUCUCGAUCUCUAAAAUCG'),
  id='AF035635.1/619-641', name='AF035635.1',
  description='AF035635.1/619-641', dbxrefs=[]),
  SeqRecord(seq=Seq('UGAGUUCUCUAUCUCUAAAAUCG'), id='M24804.1/82-104', name='M24804.1', description='M24804.1/82-104',
  dbxrefs=[]),
  SeqRecord(seq=Seq('UAAGUUCUCGAUCUUUAAAAUCG'), id='J04373.1/6212-6234', name='J04373.1',
  description='J04373.1/6212-6234', dbxrefs=[]),
  SeqRecord(seq=Seq('UAAGUUCUCGAUCUCUAAAAUCG'), id='M24803.1/1-23', name='M24803.1', description='M24803.1/1-23',
  dbxrefs=[])],  'annotations': {}} {<strong>'secondary_structure'</strong>:
  '.AAA....&lt;&lt;&lt;&lt;aaa....&gt;&gt;&gt;&gt;'}</p>
  </blockquote>
  <p>The reason <code>vars(alignment)</code> is required is to internally construct an alignment object in Stockholm format, here's the first part of the skeleton 'pseudo-code',</p>
  <pre><code>from Bio.Align import MultipleSeqAlignment
  from Bio.SeqRecord import SeqRecord
  
  outAlign = MultipleSeqAlignment([])
  record in alignment:
      name = record.id 
      myseq = record.seq
      sr = SeqRecord(Seq(name), myseq, '', '')
      outAlign.append(sr)
  </code></pre>
  <p>The above code constructs a basic alignment, the bit thats missing is to add the following key/value to the <code>alignment</code> object of <code>{'secondary_structure': '.AAA....&lt;&lt;&lt;&lt;aaa....&gt;&gt;&gt;&gt;'}</code>.</p>
  <p>Its complicated, but you get use to it.</p>
  <hr />
  <p><a href="https://github.com/biopython/biopython/blob/master/Bio/AlignIO/StockholmIO.py" rel="nofollow noreferrer">https://github.com/biopython/biopython/blob/master/Bio/AlignIO/StockholmIO.py</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-08-24T21:25:06.087Z
Answers.LastActivityDate: 2022-09-01T20:05:08.297Z
Answers.LastEditDate: 2022-09-01T20:05:08.297Z
Answers.UserDisplayName: M__
Body: <p>Is there a sequence format allowing multiple sequences to be grouped  across a flat file? The specific application is stacking separate nucleotide sequences against a given RNA secondary structure.</p>
  <p>To rephrase the same question I'm looking to stack viral nucleotide data in different groups across a single flat file. FastaQ provides an example:</p>
  <p>Traditional fastQ,</p>
  <pre><code>@sequence_name
  AGCTGAT
  +
  ...(quality score)
  
  @sequence2
  AGCTGAC
  +
  ...(quality score)
  </code></pre>
  <p>What I'm looking for is a format,</p>
  <pre><code>@Gp1_virus1-2-3
  AGACGTAG
  +
  AGAGACAA
  +
  AGAGACAT
  
  @Gp2_virus1-2
  AGATGTAG
  +
  AGATACAA
  </code></pre>
  <p>In the above example there are two groups comprising 3 (group 1) and 2 (group 2)  individual sequences (viruses). Here 'group' represents different structures.</p>
  <p>Stockholme format has potential. My understanding of this format, which is basic, is it stacks a sequence, together with structural information. What is sought is stacking multiple sequences and then a structure format. Any clarification would help.</p>
  <p>There are multiple XML formats, which may be relevant.</p>
  <p>Just to mention a 'flat file' is needed to allow manual inspection. Dumping JSON is a solution, but isn't friendly for optical inspection.</p>
  <hr />
  <p><strong>Application</strong>
  The immediate downstream application is structure/sequence viewers. I think at least one. viewer will accept Stockholme 1.0 format. rFAM is very similar (@terdon's comment)</p>
  <p><strong>Summarised Question</strong> The question could be rephrased as to whether multiple RNA families can be stacked inside one Stockholme 1.0 or else rFAM format?</p>
  <p>Some of the members here are likely familiar with <strong>pFAM</strong>. So could multiple protein structures be stored and retrieved inside one <strong>pFAM</strong> file.</p>
  <p>@gringer's contribution is very welcome.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-08-22T11:38:45.400Z
Comments.Text: What is this for? I mean, since you don't seem to need something that will be understood by a specific downstream program, can't you just define your own format?
Comments.UserDisplayName: terdon
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-08-22T15:00:19.460Z
Comments.Text: Thanks, @terdon I've updated the post. A hybrid solution may be needed (last resort).
Comments.UserDisplayName: M__
Tags: <fastq><file-formats><format-conversion><secondary-structure>
Title: Sequence format: grouping sequences within a flat file
ThreadText: # Sequence format: grouping sequences within a flat file
  
  (Asked by: username_0 on 2022-08-19)
  
  Is there a sequence format allowing multiple sequences to be grouped across a flat file? The specific application is stacking separate nucleotide sequences against a given RNA secondary structure.
  
  To rephrase the same question I'm looking to stack viral nucleotide data in different groups across a single flat file. FastaQ provides an example:
  
  Traditional fastQ,
  
  ```
  @sequence_name
  AGCTGAT
  +
  ...(quality score)
  
  @sequence2
  AGCTGAC
  +
  ...(quality score)
  
  ```
  
  What I'm looking for is a format,
  
  ```
  @Gp1_virus1-2-3
  AGACGTAG
  +
  AGAGACAA
  +
  AGAGACAT
  
  @Gp2_virus1-2
  AGATGTAG
  +
  AGATACAA
  
  ```
  
  In the above example there are two groups comprising 3 (group 1) and 2 (group 2) individual sequences (viruses). Here 'group' represents different structures.
  
  Stockholme format has potential. My understanding of this format, which is basic, is it stacks a sequence, together with structural information. What is sought is stacking multiple sequences and then a structure format. Any clarification would help.
  
  There are multiple XML formats, which may be relevant.
  
  Just to mention a 'flat file' is needed to allow manual inspection. Dumping JSON is a solution, but isn't friendly for optical inspection.
  
  ---
  
  **Application**
  The immediate downstream application is structure/sequence viewers. I think at least one. viewer will accept Stockholme 1.0 format. rFAM is very similar (@username_1's comment)
  
  **Summarised Question** The question could be rephrased as to whether multiple RNA families can be stacked inside one Stockholme 1.0 or else rFAM format?
  
  Some of the members here are likely familiar with **pFAM**. So could multiple protein structures be stored and retrieved inside one **pFAM** file.
  
  @username_2's contribution is very welcome.
  
  **Tags:** fastq, file-formats, format-conversion, secondary-structure
  
  > **Comments:**
  > **username_1**: What is this for? I mean, since you don't seem to need something that will be understood by a specific downstream program, can't you just define your own format?  
  > **username_0**: Thanks, @username_1 I've updated the post. A hybrid solution may be needed (last resort).  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2022-08-20. Score: 2 | Accepted answer
  
  An unmapped SAM file format allows for read groups. You can use `samtools import` to create unmapped SAM files from FASTQ files:
  
  ```
  samtools import -R virus-1-2 in12.fq > 12_reads.sam
  samtools import -R virus-1-2-3 in123.fq > 123_reads.sam
  samtools merge out.sam 12_reads.sam 123_reads.sam
  
  ```
  
  > **Comments:**
  > **username_0**: Perfect, thank you. It can be manipulated via Biopython `SamtoolsCalmdCommandline`  
  
  ---
  
  ### Answer by username_0 on 2022-08-24. Score: 0
  
  I'm awarding 'solved' to @username_2, because thats the context of the question.
  
  The best solution I could find is stacking `Stockholm (sth)` formats together. In this example the RNA families are associated with one structure:
  
  ```
  # STOCKHOLM 1.0
  #=GF ID    UPSK
  AF035635.1/619-641             UGAGUUCUCGAUCUCUAAAAUCG
  M24804.1/82-104                UGAGUUCUCUAUCUCUAAAAUCG
  J04373.1/6212-6234             UAAGUUCUCGAUCUUUAAAAUCG
  M24803.1/1-23                  UAAGUUCUCGAUCUCUAAAAUCG
  #=GC SS_cons                   .AAA....<<<<aaa....>>>>
  //
  
  ```
  
  `Biopython` allows `sth` format to be input.
  
  ```
  from Bio import AlignIO
  from pathlib import Path
  
  file = 'test.sth'
  path = '/Volumes/mypath/testSth'
  alignment = AlignIO.read(open(Path(path, file)), "stockholm")
  print("Alignment length %i" % alignment.get_alignment_length())
  print (alignment.column_annotations['secondary_structure'])
  
  ```
  
  Output,
  
  ```
  Alignment length 23 
  .AAA....<<<<aaa....>>>> # this is the RNA structure 
  
  ```
  
  If we look the `alignment` object via
  
  ```
  from pprint import pprint
  pprint(vars(alignment)) # or via print(alignment.__dict__)
  
  ```
  
  The output
  
  > {'\_annotations': {}, '**\_per\_col\_annotations**': {**'secondary\_structure'**:
  > '.AAA....<<<<aaa....>>>>'},  
  > '\_records':
  > [SeqRecord(seq=Seq('UGAGUUCUCGAUCUCUAAAAUCG'),
  > id='AF035635.1/619-641', name='AF035635.1',
  > description='AF035635.1/619-641', dbxrefs=[]),
  > SeqRecord(seq=Seq('UGAGUUCUCUAUCUCUAAAAUCG'), id='M24804.1/82-104', name='M24804.1', description='M24804.1/82-104',
  > dbxrefs=[]),
  > SeqRecord(seq=Seq('UAAGUUCUCGAUCUUUAAAAUCG'), id='J04373.1/6212-6234', name='J04373.1',
  > description='J04373.1/6212-6234', dbxrefs=[]),
  > SeqRecord(seq=Seq('UAAGUUCUCGAUCUCUAAAAUCG'), id='M24803.1/1-23', name='M24803.1', description='M24803.1/1-23',
  > dbxrefs=[])], 'annotations': {}} {**'secondary\_structure'**:
  > '.AAA....<<<<aaa....>>>>'}
  
  The reason `vars(alignment)` is required is to internally construct an alignment object in Stockholm format, here's the first part of the skeleton 'pseudo-code',
  
  ```
  from Bio.Align import MultipleSeqAlignment
  from Bio.SeqRecord import SeqRecord
  
  outAlign = MultipleSeqAlignment([])
  record in alignment:
      name = record.id 
      myseq = record.seq
      sr = SeqRecord(Seq(name), myseq, '', '')
      outAlign.append(sr)
  
  ```
  
  The above code constructs a basic alignment, the bit thats missing is to add the following key/value to the `alignment` object of `{'secondary_structure': '.AAA....<<<<aaa....>>>>'}`.
  
  Its complicated, but you get use to it.
  
  ---
  
  <https://github.com/biopython/biopython/blob/master/Bio/AlignIO/StockholmIO.py>
  
  
  ---
  
  