Answers.Body: <p>Presuming we consider "human readable", "easily parsable", and "quickly queryable" to be objectively good qualities (and if not, I worry for the future):</p>
  
  <ol>
  <li>Text-based: It's absurdly common to want to use <code>grep</code> or <code>awk</code> on annotations. Sure, one could make variants of these that are binary-format aware, but why reinvent the wheel. Of course text files don't innately allow region-based queries of their contents, so on to point 2...
  
  <ul>
  <li>This should further explicitly be line-based. Fields would need to be tab-separated (ideally it'd use the ascii record separator character instead, but I fear that ship has long ago sailed).</li>
  </ul></li>
  <li>A strictly defined line order: This is one of my personal pet-peeves about BED/GFF/GTF files. If you're going to make a text-based annotation format, everyone would end up ahead if said format made explicit that it should be sorted. This then allows things like tabix to be used so the "query a region" problem is then solved. But I would go further than that. The issue with things like GFF is that there are multiple inter-dependent lines and there's no strict rule about whether a parent line absolutely must come before a child line. This just makes implementing things a nightmare and more often than not a randomly ordered file will just break tools. Since GTF or GFF-style interdependent lines likely be how any annotation format works, this ordering between lines with the same start position should be made explicit in the format.</li>
  </ol>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-06-08T08:40:22.937Z
Answers.Text: Regarding point 1, I would add a "line-based" further requirement, not just "text-based". It is also easier to manipulate if all the information is organized in tab-separated columns, but this may reduce the flexibility compared to what is currently allowed within the attributes column of gff format.
Answers.UserDisplayName: bli
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-06-08T08:44:36.700Z
Answers.Text: @bli Excellent suggestion! I've made a note of this. Flexibility is a double-edged sword; once you can do almost anything you can't reliably depend on a format :(
Answers.UserDisplayName: Devon Ryan
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-06-08T08:49:25.083Z
Answers.Text: I hadn't realized there existed a record separator ASCII character. I think tab has the merit of making files more human readable (with current text-based computing tools).
Answers.UserDisplayName: bli
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-06-08T14:20:53.163Z
Answers.Text: I’m honestly not sure whether human readable is an objective criterion for goodness. In fact, it seems to get in the way so there’s *at least* an argument to be made for a trade-off.
Answers.UserDisplayName: Konrad Rudolph
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-06-08T14:24:35.563Z
Answers.Text: @KonradRudolph Sure, all of the tabix work-arounds can be considered bandages over the "human readable" part.
Answers.UserDisplayName: Devon Ryan
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-06-08T07:18:13.830Z
Answers.LastActivityDate: 2017-06-08T08:50:10.537Z
Answers.LastEditDate: 2017-06-08T08:50:10.537Z
Answers.UserDisplayName: Devon Ryan
Answers.Body: <p>I quite like BED and GFF3 (I don't like GTF/GFF2, though). As text-based formats, I don't think they leave us much room for improvement. Anyway, if you want a new format, here is one. The following is a hybrid between GFF3 and BED. It is a TAB-delimited text-based format with the following fields:</p>
  
  <ol>
  <li>chr, required</li>
  <li>start (0-based), required</li>
  <li>end, required</li>
  <li>strand</li>
  <li>ID</li>
  <li>type (see below)</li>
  </ol>
  
  <p>Like BED, only the first 3 columns are required, the rest are optional. What is different starts with the "type" field as in GFF. This "type", when present, defines the columns following it. For example, if type==coding, we could have cdsStart, cdsEnd, blockCount, blockSizes and blockStarts like BED; if type==exon, column 7 could be a "phase", indicating the phase of the first base. This way, "type" makes this format highly extensible while still relatively easy to parse in comparison to using optional tags all the way. In addition, we may have semi-colon-separated "key"="value" pairs at the end of each line as in GFF3. Example:</p>
  
  <pre><code>chr1   10000  50000  -  x1 coding   10100   40800  2  1000,2000   10000,48000
  chr1   10000  11000  -  *  exon     *       foo1=bar1;foo2=bar2
  chr1   48000  50000  -  *  exon     2
  chr2   10000  50000
  </code></pre>
  
  <p>The above only gives a bare bone of the format. There are subtle questions such as: 1) the uniqueness/scope of ID; 2) whether to have parent ID as a fixed field or a type-specific field; 3) where to put display name; 4) sorting order. These are also important in practice.</p>
  
  <p>PS: I like SQL a lot and think it should get used more often in bioinformatics, but I don't think it replaces text formats completely. Formats are useful for serialization and data transfer. They require less skill to work with and less software/hardware resource to deploy. Carefully engineered binary representations of formats can be much more efficient in a lot of use cases.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-06-08T15:49:51.480Z
Answers.LastActivityDate: 2017-06-08T15:49:51.480Z
Answers.UserDisplayName: user172818
Answers.Body: <p>The problem is that GFF, fundamentally, is a <a href="https://en.wikipedia.org/wiki/Relational_model" rel="nofollow noreferrer">relational</a> format: it provides tags that relate individual rows via one-to-many relationships (e.g. gene–exon). This indirectly highlights the second complication: individual rows have different types, and therefore store different attributes in the 9th column.</p>
  
  <p>Over the last few decades (!), we have accumulated a wealth of theory and tools to work with this kind of data. And the usual solution is to create a database schema for a relational database, and to use database drivers and database query languages (e.g. SQL, but increasingly also data-relational mappers such as LINQ and dplyr).</p>
  
  <p>Using a text-based format is attractive for many reasons that Devon mentions, but it is fundamentally at odds with a lot of the theory and tools for relational data. This creates an <a href="https://en.wikipedia.org/wiki/Object-relational_impedance_mismatch" rel="nofollow noreferrer">impedance mismatch</a>.</p>
  
  <p>I’m convinced that, in the long run, the solution is going to be to revert to using relational databases for complex annotations. I say “revert” even though these databases already exist, they are simply often ignored in bioinformaticians’ day-to-day work (<em>I</em> never use them). Because this is objectively the best technical solution, and we have the research to back this up.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-06-08T15:06:13.250Z
Answers.Text: I guess that as long as one only does very simple things with the annotation information, one (=me) is happy with line-based format that can be processed with common command-line tools. However, your point of view is interesting and you seem to have a much better understanding of these issues than me. I'll try to get at least an idea of what "impedance mismatch" is.
Answers.UserDisplayName: bli
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-06-08T15:10:45.870Z
Answers.Text: Totally agree, flat files are not the way forwards for complex data structures, although I am guilty of this. Any thoughts on SQL vs NoSQL?
Answers.UserDisplayName: Chris_Rands
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-06-08T15:56:38.920Z
Answers.Text: @Chris_Rands I have no experience working with NoSQL but since the data here is specifically *relational*, I don’t think NoSQL would offer any specific benefits.
Answers.UserDisplayName: Konrad Rudolph
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-07-24T13:06:26.847Z
Answers.Text: Seems more like "nosql" rather than relational in some case. Just look at the mess that Chado is, with hundreds of "linking tables"
Answers.UserDisplayName: Colin D
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-06-08T14:32:07.633Z
Answers.LastActivityDate: 2017-06-08T14:32:07.633Z
Answers.UserDisplayName: Konrad Rudolph
Answers.Body: <p>Just throwing this out there, as all the best stuff has already been said, but why does one need to specify a strict specification for a bioinformatic data format? As you say, what typically happens is all the action ends up in the optional fields.</p>
  
  <p>There's a lot to be said for the libertarian values of just letting people figure it out on their own. Take optional fields in the BAM tag spec. for example. Here you can have your own tags, but they must start with an "X", a "Y" or a "Z" and be followed by "[A-Za-z0-9]". Why? Why can't people use their own tag names like "read is unique" or "edit distance to genome" or whatever they want. Are we not to be trusted with the power of naming things and recalling things arbitrarily? The result is one must look up the tag name's actual meaning in the publication of the tool that produced it, or ask on certain forums, etc. And this is assuming one knows the tool that produced the reads - if not then who knows wtf 'XT' stands for.</p>
  
  <p>Essentially, downstream readers of optional fields already trust that 'XT' is what it thinks it is. So if we're happy to use optional fields in any capacity, why stop at fields?</p>
  
  <p>Extend this logic to the columns of the data format. Let users determine column names. You may once in a blue moon come across a file with the chromosome column as "chr" rather than "chromosome", but most of the time you'll be looking for "read is quantum-radioactive", and detecting and fixing naming-errors is not going to be as much of a problem as using a dataformat that does not store what it is you want to store. Or worse, it can store it, but in a really illogical way that results in everyone who uses the dataformat to have to ask at least 3 questions here or on other forums before they understand whats really going on.</p>
  
  <p>So you end up with a general solution to the problem of storing tabular information in a compatible way across many different types of software, also known as SQL (or any other kind of general-purpose database such as REDIS, Neo4J, Numpy, etc). In fact, does it even matter what the datastore is, so long as it's tabular and has the "chromosome" and "position" values for each item?</p>
  
  <p>TL;DR - We will not think of tomorrow's best data format today, because the nature of tomorrow's data is not yet known. Less policing in this area would most likely result in more robust software, where nothing is taken for granted and no assumptions about the data can be made until the schema has been parsed.</p>
  
  <p>In truth, the only reason we think we need to specify our dataformats is because we know if we didn't, other people will do something we didn't think of - and we, i think falsely, assume that will be bad.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-06-08T22:09:13.897Z
Answers.LastActivityDate: 2017-06-09T01:59:00.970Z
Answers.LastEditDate: 2017-06-09T01:59:00.970Z
Answers.UserDisplayName: John Longinotto
Body: <p>Bashing file formats is a favorite pastime in bioinformatics, and annotation file formats such as GFF and BED seem to get special attention. A lot of this frustration stems from community's shockingly inconsistent adherence to specifications and conventions, but there are also some (dare I say objectively) problematic design choices in each of these formats.</p>
  
  <ul>
  <li><p>GFF (and its more common derivatives GTF and GFF3) use 1-based closed interval notation, which optimizes for human comprehension but is far inferior to 0-based half-open interval notation (such as used by BED) for computations involving interval arithmetic.</p></li>
  <li><p>Although BED and GTF were designed for very specific use cases (visualization and gene prediction, respectively), they have been used and abused in a much wider set of contexts. For example, the BED fields related to the <strong>thick part</strong> are irrelevant if you're not plotting them in a genome browser.</p></li>
  <li><p>BED supports a single level of feature decomposition (a feature can be broken up into blocks). GTF supports two levels (exons grouped by transcript_id, transcripts grouped by gene_id). In contrast, GFF3 supports an arbitrary number of levels, and uses parent/child relationships defined by <code>ID</code> and <code>Parent</code> attributes to declare a directed acyclic graph of features.</p></li>
  <li><p>Data that does not fit into mandatory pre-defined fields must be relegated to optional fields or free-form attribute key/value pairs. While this flexibility is powerful, a common complaint is that "all the action" happens in these optional/free-form fields.</p></li>
  <li><p>There is a dearth of validation tools, and those that do exist focus primarily on validating syntax and not semantics. To use an aging analogy, it's one thing to say an XML file is valid, but it's completely different to validate it against a schema. There are essentially no widely used tools that do the latter for annotation files.</p></li>
  </ul>
  
  <p>If we were tasked with creating a new annotation format, and if we were guaranteed the resources needed to develop it, and interest and wide adoption from the wider community (one can dream!), what design criteria should be considered in the development of this new format? <strong>What, if anything, makes an objectively good annotation data format?</strong></p>
  
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-06-08T11:28:46.107Z
Comments.Text: Are you only asking about a format describing genomic features? "Annotation" is a very broad term but it looks like you're only considering genomic regions here or, at least, things that have i) a defined "region" and ii) a defined "function". That would still exclude phenotype annotations for proteins or GI annotations for genes etc. Could you [edit] and clarify what kind of "annotations" you are considering?
Comments.UserDisplayName: terdon
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-07-24T13:16:01.410Z
Comments.Text: The BED concept of autoSql is a pretty nice feature of an annotation format and allows for a lot of extensibility. The concept of feature hierarchy is still basically single level though
Comments.UserDisplayName: Colin D
Tags: <file-formats><bed><gff3><sequence-annotation>
Title: Annotation format design
ThreadText: # Annotation format design
  
  (Asked by: username_0 on 2017-06-08)
  
  Bashing file formats is a favorite pastime in bioinformatics, and annotation file formats such as GFF and BED seem to get special attention. A lot of this frustration stems from community's shockingly inconsistent adherence to specifications and conventions, but there are also some (dare I say objectively) problematic design choices in each of these formats.
  
  * GFF (and its more common derivatives GTF and GFF3) use 1-based closed interval notation, which optimizes for human comprehension but is far inferior to 0-based half-open interval notation (such as used by BED) for computations involving interval arithmetic.
  * Although BED and GTF were designed for very specific use cases (visualization and gene prediction, respectively), they have been used and abused in a much wider set of contexts. For example, the BED fields related to the **thick part** are irrelevant if you're not plotting them in a genome browser.
  * BED supports a single level of feature decomposition (a feature can be broken up into blocks). GTF supports two levels (exons grouped by transcript\_id, transcripts grouped by gene\_id). In contrast, GFF3 supports an arbitrary number of levels, and uses parent/child relationships defined by `ID` and `Parent` attributes to declare a directed acyclic graph of features.
  * Data that does not fit into mandatory pre-defined fields must be relegated to optional fields or free-form attribute key/value pairs. While this flexibility is powerful, a common complaint is that "all the action" happens in these optional/free-form fields.
  * There is a dearth of validation tools, and those that do exist focus primarily on validating syntax and not semantics. To use an aging analogy, it's one thing to say an XML file is valid, but it's completely different to validate it against a schema. There are essentially no widely used tools that do the latter for annotation files.
  
  If we were tasked with creating a new annotation format, and if we were guaranteed the resources needed to develop it, and interest and wide adoption from the wider community (one can dream!), what design criteria should be considered in the development of this new format? **What, if anything, makes an objectively good annotation data format?**
  
  **Tags:** file-formats, bed, gff3, sequence-annotation
  
  > **Comments:**
  > **username_1**: Are you only asking about a format describing genomic features? "Annotation" is a very broad term but it looks like you're only considering genomic regions here or, at least, things that have i) a defined "region" and ii) a defined "function". That would still exclude phenotype annotations for proteins or GI annotations for genes etc. Could you [edit] and clarify what kind of "annotations" you are considering?  
  > **username_2**: The BED concept of autoSql is a pretty nice feature of an annotation format and allows for a lot of extensibility. The concept of feature hierarchy is still basically single level though  
  
  ---
  
  ## 4 answers
  
  ### Answer by username_3 on 2017-06-08. Score: 4
  
  Presuming we consider "human readable", "easily parsable", and "quickly queryable" to be objectively good qualities (and if not, I worry for the future):
  
  1. Text-based: It's absurdly common to want to use `grep` or `awk` on annotations. Sure, one could make variants of these that are binary-format aware, but why reinvent the wheel. Of course text files don't innately allow region-based queries of their contents, so on to point 2...
     * This should further explicitly be line-based. Fields would need to be tab-separated (ideally it'd use the ascii record separator character instead, but I fear that ship has long ago sailed).
  2. A strictly defined line order: This is one of my personal pet-peeves about BED/GFF/GTF files. If you're going to make a text-based annotation format, everyone would end up ahead if said format made explicit that it should be sorted. This then allows things like tabix to be used so the "query a region" problem is then solved. But I would go further than that. The issue with things like GFF is that there are multiple inter-dependent lines and there's no strict rule about whether a parent line absolutely must come before a child line. This just makes implementing things a nightmare and more often than not a randomly ordered file will just break tools. Since GTF or GFF-style interdependent lines likely be how any annotation format works, this ordering between lines with the same start position should be made explicit in the format.
  
  > **Comments:**
  > **username_4**: Regarding point 1, I would add a "line-based" further requirement, not just "text-based". It is also easier to manipulate if all the information is organized in tab-separated columns, but this may reduce the flexibility compared to what is currently allowed within the attributes column of gff format.  
  > **username_3**: @username_4 Excellent suggestion! I've made a note of this. Flexibility is a double-edged sword; once you can do almost anything you can't reliably depend on a format :(  
  > **username_4**: I hadn't realized there existed a record separator ASCII character. I think tab has the merit of making files more human readable (with current text-based computing tools).  
  > **username_5**: I’m honestly not sure whether human readable is an objective criterion for goodness. In fact, it seems to get in the way so there’s *at least* an argument to be made for a trade-off.  
  > **username_3**: @username_5 Sure, all of the tabix work-arounds can be considered bandages over the "human readable" part.  
  
  ---
  
  ### Answer by username_6 on 2017-06-08. Score: 4
  
  I quite like BED and GFF3 (I don't like GTF/GFF2, though). As text-based formats, I don't think they leave us much room for improvement. Anyway, if you want a new format, here is one. The following is a hybrid between GFF3 and BED. It is a TAB-delimited text-based format with the following fields:
  
  1. chr, required
  2. start (0-based), required
  3. end, required
  4. strand
  5. ID
  6. type (see below)
  
  Like BED, only the first 3 columns are required, the rest are optional. What is different starts with the "type" field as in GFF. This "type", when present, defines the columns following it. For example, if type==coding, we could have cdsStart, cdsEnd, blockCount, blockSizes and blockStarts like BED; if type==exon, column 7 could be a "phase", indicating the phase of the first base. This way, "type" makes this format highly extensible while still relatively easy to parse in comparison to using optional tags all the way. In addition, we may have semi-colon-separated "key"="value" pairs at the end of each line as in GFF3. Example:
  
  ```
  chr1   10000  50000  -  x1 coding   10100   40800  2  1000,2000   10000,48000
  chr1   10000  11000  -  *  exon     *       foo1=bar1;foo2=bar2
  chr1   48000  50000  -  *  exon     2
  chr2   10000  50000
  
  ```
  
  The above only gives a bare bone of the format. There are subtle questions such as: 1) the uniqueness/scope of ID; 2) whether to have parent ID as a fixed field or a type-specific field; 3) where to put display name; 4) sorting order. These are also important in practice.
  
  PS: I like SQL a lot and think it should get used more often in bioinformatics, but I don't think it replaces text formats completely. Formats are useful for serialization and data transfer. They require less skill to work with and less software/hardware resource to deploy. Carefully engineered binary representations of formats can be much more efficient in a lot of use cases.
  
  
  ---
  
  ### Answer by username_5 on 2017-06-08. Score: 4
  
  The problem is that GFF, fundamentally, is a [relational](https://en.wikipedia.org/wiki/Relational_model) format: it provides tags that relate individual rows via one-to-many relationships (e.g. gene–exon). This indirectly highlights the second complication: individual rows have different types, and therefore store different attributes in the 9th column.
  
  Over the last few decades (!), we have accumulated a wealth of theory and tools to work with this kind of data. And the usual solution is to create a database schema for a relational database, and to use database drivers and database query languages (e.g. SQL, but increasingly also data-relational mappers such as LINQ and dplyr).
  
  Using a text-based format is attractive for many reasons that username_3 mentions, but it is fundamentally at odds with a lot of the theory and tools for relational data. This creates an [impedance mismatch](https://en.wikipedia.org/wiki/Object-relational_impedance_mismatch).
  
  I’m convinced that, in the long run, the solution is going to be to revert to using relational databases for complex annotations. I say “revert” even though these databases already exist, they are simply often ignored in bioinformaticians’ day-to-day work (*I* never use them). Because this is objectively the best technical solution, and we have the research to back this up.
  
  > **Comments:**
  > **username_4**: I guess that as long as one only does very simple things with the annotation information, one (=me) is happy with line-based format that can be processed with common command-line tools. However, your point of view is interesting and you seem to have a much better understanding of these issues than me. I'll try to get at least an idea of what "impedance mismatch" is.  
  > **username_7**: Totally agree, flat files are not the way forwards for complex data structures, although I am guilty of this. Any thoughts on SQL vs NoSQL?  
  > **username_5**: @username_7 I have no experience working with NoSQL but since the data here is specifically *relational*, I don’t think NoSQL would offer any specific benefits.  
  > **username_2**: Seems more like "nosql" rather than relational in some case. Just look at the mess that Chado is, with hundreds of "linking tables"  
  
  ---
  
  ### Answer by username_8 on 2017-06-08. Score: 0
  
  Just throwing this out there, as all the best stuff has already been said, but why does one need to specify a strict specification for a bioinformatic data format? As you say, what typically happens is all the action ends up in the optional fields.
  
  There's a lot to be said for the libertarian values of just letting people figure it out on their own. Take optional fields in the BAM tag spec. for example. Here you can have your own tags, but they must start with an "X", a "Y" or a "Z" and be followed by "[A-Za-z0-9]". Why? Why can't people use their own tag names like "read is unique" or "edit distance to genome" or whatever they want. Are we not to be trusted with the power of naming things and recalling things arbitrarily? The result is one must look up the tag name's actual meaning in the publication of the tool that produced it, or ask on certain forums, etc. And this is assuming one knows the tool that produced the reads - if not then who knows wtf 'XT' stands for.
  
  Essentially, downstream readers of optional fields already trust that 'XT' is what it thinks it is. So if we're happy to use optional fields in any capacity, why stop at fields?
  
  Extend this logic to the columns of the data format. Let users determine column names. You may once in a blue moon come across a file with the chromosome column as "chr" rather than "chromosome", but most of the time you'll be looking for "read is quantum-radioactive", and detecting and fixing naming-errors is not going to be as much of a problem as using a dataformat that does not store what it is you want to store. Or worse, it can store it, but in a really illogical way that results in everyone who uses the dataformat to have to ask at least 3 questions here or on other forums before they understand whats really going on.
  
  So you end up with a general solution to the problem of storing tabular information in a compatible way across many different types of software, also known as SQL (or any other kind of general-purpose database such as REDIS, Neo4J, Numpy, etc). In fact, does it even matter what the datastore is, so long as it's tabular and has the "chromosome" and "position" values for each item?
  
  TL;DR - We will not think of tomorrow's best data format today, because the nature of tomorrow's data is not yet known. Less policing in this area would most likely result in more robust software, where nothing is taken for granted and no assumptions about the data can be made until the schema has been parsed.
  
  In truth, the only reason we think we need to specify our dataformats is because we know if we didn't, other people will do something we didn't think of - and we, i think falsely, assume that will be bad.
  
  
  ---
  
  