Answers.Body: <p>BioJS has capacity to read and write GFF as streams, it might have something of interest such as this: <a href="https://github.com/biojs-io/biojs-io-gff" rel="nofollow noreferrer">https://github.com/biojs-io/biojs-io-gff</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-09-13T16:49:19.870Z
Answers.Text: Oh neat! Looks like this implementation converts GFF to its own internal JSON representation.
Answers.UserDisplayName: juniper-
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-09-13T11:43:07.743Z
Answers.LastActivityDate: 2019-09-13T12:01:42.363Z
Answers.LastEditDate: 2019-09-13T12:01:42.363Z
Answers.UserDisplayName: Jay Moore
Answers.Body: <p>I've written a Python library called <a href="https://github.com/SACGF/pyreference" rel="nofollow noreferrer">PyReference</a> which converts a GTF / GFF (RefSeq and Ensembl) into a gzipped JSON file.</p>
  <p>You can just use this if you like but there's also a Python wrapper around the JSON, which allows you to write genomics code more naturally.</p>
  <p>Python isn't known for being fast but the library function for reading a JSON file is highly optimised - the following takes less than 4 seconds on my laptop:</p>
  <pre><code>import numpy as np
  import pyreference
  
  reference = pyreference.Reference()
  
  my_gene_ids = [&quot;MSN&quot;, &quot;GATA2&quot;, &quot;ZEB1&quot;]
  for gene in reference[my_gene_ids]:
      average_length = np.mean([t.length for t in gene.transcripts])
      print(&quot;%s average length = %.2f&quot; % (gene, average_length))
      print(gene.iv)
      for transcript in gene.transcripts:
          if transcript.is_coding:
              threep_utr = transcript.get_3putr_sequence()
              print(&quot;%s end of 3putr: %s&quot; % (transcript.get_id(), threep_utr[-20:]))
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-16T03:49:33.020Z
Answers.Text: This looks promising, thank you! It would be even better if there was an API reference linked to in the GitHub repo ;-)
Answers.UserDisplayName: juniper-
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-14T04:30:48.027Z
Answers.LastActivityDate: 2022-01-14T04:30:48.027Z
Answers.UserDisplayName: David Lawrence
Answers.Body: <p><a href="https://github.com/ekg/jvcf" rel="nofollow noreferrer">JVCF</a> is a JSON schema for describing genetic variants. <a href="https://jalview.github.io/biojson/" rel="nofollow noreferrer">BioJSON</a> is a JSON schema for multiple sequence alignments. The output of <a href="https://mygene.info/" rel="nofollow noreferrer">mygene.info</a>'s gene annotation web service is JSON-formatted. </p>
  
  <p>You might build one of your own schemas from any or all of these, as inspiration for structure and content. </p>
  
  <p>Consider that most genomics formats are historically tabular, because they are not hierarchical, because they originated prior to the advent of JavaScript as a common programming language (and subsequent use of JSON to serialize data), and because a lot of stream-oriented UNIX tools read in data and process it in ways that are natural to tabular structure.</p>
  
  <p>Some tabular results can be hierarchical, but use inefficiencies in field values to apply hierarchy. JSON is inefficient for storing tabular data, but it is absolutely wonderful for hierarchical data. This difference is perhaps worth considering.</p>
  
  <p>One could think of storing a gene as a master object containing various haplotypes, exons, introns, features, associated variants, associated TF binding sites, gene ontologies, etc. Ordering or prioritization of such items would be possible with the use of sorted lists within a gene object. The format is as extensible as it is a matter of adding a property, list, or object to the schema. </p>
  
  <p>To gain adoption or "buy-in", build tools which publish it to users, and tools that consume what users get in order to do things, along with conversion and validation utilities.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-09-13T16:42:32.903Z
Answers.Text: Thanks for the detailed answer! I think you hit the nail on the head with tabular vs. hierarchical data. Formats like GFF, Genbank and genePred store hierarchical data in a tabular format which makes it difficult to parse. Contigs, genes, mRNAs, CDSs, introns and exons have a very natural organization that may be well suited to a JSON-like structure.
Answers.UserDisplayName: juniper-
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-09-12T20:15:27.040Z
Answers.LastActivityDate: 2019-09-14T09:40:10.650Z
Answers.LastEditDate: 2019-09-14T09:40:10.650Z
Answers.UserDisplayName: Alex Reynolds
Body: <p>There's a variety of formats for storing genomic features (e.g. Genbank, GFF, GTF, Bedfile) but all of the ones that I'm familiar with use either a custom format (Genbank) or a CSV with defined column conventions and space for additional data (GFF, GTF, Bedfile).</p>
  
  <p>Is there a feature format that is JSON based and adheres to a defined schema?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-09-12T15:57:36.890Z
Comments.Text: It is a nice question, but why do you expect JSON schema for this type of data What would be its advantages over other file formats?
Comments.UserDisplayName: llrs
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-09-12T16:03:24.943Z
Comments.Text: Easy to validate, easy to understand, enforcement of conventions. I find that with bed or gff files I'm constantly either going back to the definition to find out what the columns are or trying to figure them out based on their content. This is especially annoying with a multi-column tsv file that has missing values for columns (as can happen in the genePred format).
  
  Some file types (e.g. GFF) allow for additional fields whose names are not enforced. So one file could have `geneName` and another could have `gene_name`.
Comments.UserDisplayName: juniper-
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-09-12T16:23:01.657Z
Comments.Text: Bear in mind that this is a very subjective issue. For example, I find JSON an absolute pain to work with and very hard to understand, while simple, tab-delimited text files are a pleasure to work with. Note that if a field is empty, you have `F1\tF2\t\tF4`, so empty fields are also easily dealt with.
Comments.UserDisplayName: terdon
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-09-12T16:26:53.467Z
Comments.Text: @terdon Completely agree on the subjectivity of the issue. I'm not trying to convert anybody. Just curious if such a format exists.
Comments.UserDisplayName: juniper-
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-09-12T16:28:47.553Z
Comments.Text: Would you be interested in solutions that somehow convert gff to json? I'm sure there are tools or Bio[Python|Perl|R] modules for this. It would help if you could clarify exactly what genomic features you're after though. Since it's such a vague term, you kinda _need_ vague formats for it :/.
Comments.UserDisplayName: terdon
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-09-12T16:50:46.000Z
Comments.Text: @terdon Yes, I would be interested in those as well. Genomic features is indeed a broad term but there are a number of concepts that reused repeatedly across formats. `contig`, `start`, `end`, `gene`, `cds`, `mrna`, `trna` come immediately to mind but I'm sure there's plenty of others.
Comments.UserDisplayName: juniper-
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-09-12T16:57:18.740Z
Comments.Text: So you are only interested in 1-dimensional, linear features? No trans-splicing, secondary, tertiary or quaternary structure etc? Should be trivial to convert that to JSON if that's what you want, yes.
Comments.UserDisplayName: terdon
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-09-12T17:04:55.153Z
Comments.Text: Yes, only 1d linear features. No doubt about it being trivial to convert, I'm just curious if there's a format / schema people already use.
Comments.UserDisplayName: juniper-
Tags: <features><genomefeatures>
Title: Is there a JSON-based genomic feature format?
ThreadText: # Is there a JSON-based genomic feature format?
  
  (Asked by: username_0 on 2019-09-12)
  
  There's a variety of formats for storing genomic features (e.g. Genbank, GFF, GTF, Bedfile) but all of the ones that I'm familiar with use either a custom format (Genbank) or a CSV with defined column conventions and space for additional data (GFF, GTF, Bedfile).
  
  Is there a feature format that is JSON based and adheres to a defined schema?
  
  **Tags:** features, genomefeatures
  
  > **Comments:**
  > **username_1**: It is a nice question, but why do you expect JSON schema for this type of data What would be its advantages over other file formats?  
  > **username_0**: Easy to validate, easy to understand, enforcement of conventions. I find that with bed or gff files I'm constantly either going back to the definition to find out what the columns are or trying to figure them out based on their content. This is especially annoying with a multi-column tsv file that has missing values for columns (as can happen in the genePred format).  Some file types (e.g. GFF) allow for additional fields whose names are not enforced. So one file could have `geneName` and another could have `gene_name`.  
  > **username_2**: Bear in mind that this is a very subjective issue. For example, I find JSON an absolute pain to work with and very hard to understand, while simple, tab-delimited text files are a pleasure to work with. Note that if a field is empty, you have `F1\tF2\t\tF4`, so empty fields are also easily dealt with.  
  > **username_0**: @username_2 Completely agree on the subjectivity of the issue. I'm not trying to convert anybody. Just curious if such a format exists.  
  > **username_2**: Would you be interested in solutions that somehow convert gff to json? I'm sure there are tools or Bio[Python|Perl|R] modules for this. It would help if you could clarify exactly what genomic features you're after though. Since it's such a vague term, you kinda _need_ vague formats for it :/.  
  > **username_0**: @username_2 Yes, I would be interested in those as well. Genomic features is indeed a broad term but there are a number of concepts that reused repeatedly across formats. `contig`, `start`, `end`, `gene`, `cds`, `mrna`, `trna` come immediately to mind but I'm sure there's plenty of others.  
  > **username_2**: So you are only interested in 1-dimensional, linear features? No trans-splicing, secondary, tertiary or quaternary structure etc? Should be trivial to convert that to JSON if that's what you want, yes.  
  > **username_0**: Yes, only 1d linear features. No doubt about it being trivial to convert, I'm just curious if there's a format / schema people already use.  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_5 on 2019-09-12. Score: 5 | Accepted answer
  
  [JVCF](https://github.com/ekg/jvcf) is a JSON schema for describing genetic variants. [BioJSON](https://jalview.github.io/biojson/) is a JSON schema for multiple sequence alignments. The output of [mygene.info](https://mygene.info/)'s gene annotation web service is JSON-formatted.
  
  You might build one of your own schemas from any or all of these, as inspiration for structure and content.
  
  Consider that most genomics formats are historically tabular, because they are not hierarchical, because they originated prior to the advent of JavaScript as a common programming language (and subsequent use of JSON to serialize data), and because a lot of stream-oriented UNIX tools read in data and process it in ways that are natural to tabular structure.
  
  Some tabular results can be hierarchical, but use inefficiencies in field values to apply hierarchy. JSON is inefficient for storing tabular data, but it is absolutely wonderful for hierarchical data. This difference is perhaps worth considering.
  
  One could think of storing a gene as a master object containing various haplotypes, exons, introns, features, associated variants, associated TF binding sites, gene ontologies, etc. Ordering or prioritization of such items would be possible with the use of sorted lists within a gene object. The format is as extensible as it is a matter of adding a property, list, or object to the schema.
  
  To gain adoption or "buy-in", build tools which publish it to users, and tools that consume what users get in order to do things, along with conversion and validation utilities.
  
  > **Comments:**
  > **username_0**: Thanks for the detailed answer! I think you hit the nail on the head with tabular vs. hierarchical data. Formats like GFF, Genbank and genePred store hierarchical data in a tabular format which makes it difficult to parse. Contigs, genes, mRNAs, CDSs, introns and exons have a very natural organization that may be well suited to a JSON-like structure.  
  
  ---
  
  ### Answer by username_3 on 2019-09-13. Score: 1
  
  BioJS has capacity to read and write GFF as streams, it might have something of interest such as this: <https://github.com/biojs-io/biojs-io-gff>
  
  > **Comments:**
  > **username_0**: Oh neat! Looks like this implementation converts GFF to its own internal JSON representation.  
  
  ---
  
  ### Answer by username_4 on 2022-01-14. Score: 1
  
  I've written a Python library called [PyReference](https://github.com/SACGF/pyreference) which converts a GTF / GFF (RefSeq and Ensembl) into a gzipped JSON file.
  
  You can just use this if you like but there's also a Python wrapper around the JSON, which allows you to write genomics code more naturally.
  
  Python isn't known for being fast but the library function for reading a JSON file is highly optimised - the following takes less than 4 seconds on my laptop:
  
  ```
  import numpy as np
  import pyreference
  
  reference = pyreference.Reference()
  
  my_gene_ids = ["MSN", "GATA2", "ZEB1"]
  for gene in reference[my_gene_ids]:
      average_length = np.mean([t.length for t in gene.transcripts])
      print("%s average length = %.2f" % (gene, average_length))
      print(gene.iv)
      for transcript in gene.transcripts:
          if transcript.is_coding:
              threep_utr = transcript.get_3putr_sequence()
              print("%s end of 3putr: %s" % (transcript.get_id(), threep_utr[-20:]))
  
  ```
  
  > **Comments:**
  > **username_0**: This looks promising, thank you! It would be even better if there was an API reference linked to in the GitHub repo ;-)  
  
  ---
  
  