Answers.Body: <p>BioJulia maintains <a href="https://github.com/BioJulia/BioFmtSpecimens" rel="nofollow noreferrer">BioFmtSpecimens</a>. From the README:</p>
  <blockquote>
  <p>Bioinformatics is rife with formats and parsers for those formats. These parsers don't always agree on the definitions of these formats, since many lack any sort of formal standard.</p>
  <p>This repository aims to consolidate a collection of format specimens to create a unified set of data with which to test software against. Testing against the same cases is a first step towards agreeing on the details and edge cases of a format.</p>
  </blockquote>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-31T11:26:12.363Z
Answers.LastActivityDate: 2020-03-31T11:26:12.363Z
Answers.LastEditDate: 2020-06-18T08:30:41.743Z
Answers.UserDisplayName: jgreener
Answers.Body: <p>Not that I am aware. It is best to go with format specifications when coding.</p>
  
  <p>Also it may be good to look at the example files that come together with various tools performing file conversions and handling. E.g. </p>
  
  <ul>
  <li>Trimmomatic comes with few .fa files (<a href="http://www.usadellab.org/cms/?page=trimmomatic" rel="noreferrer">http://www.usadellab.org/cms/?page=trimmomatic</a>)</li>
  <li>Samtools with a toy.sam (<a href="https://github.com/lh3/samtools-legacy/blob/master/examples/toy.sam" rel="noreferrer">https://github.com/lh3/samtools-legacy/blob/master/examples/toy.sam</a>)</li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-06-02T19:44:00.670Z
Answers.LastActivityDate: 2017-06-02T19:58:28.837Z
Answers.LastEditDate: 2017-06-02T19:58:28.837Z
Answers.UserDisplayName: olga
Answers.Body: <p>As far as I know, there is no single repository that collects all of the common data formats used in bioinformatics.  Typically, you have to go to the source to find the specifications for each format.  There are a few places that collect descriptions of file formats, though:</p>
  
  <ul>
  <li><a href="https://software.broadinstitute.org/software/igv/FileFormats" rel="nofollow noreferrer">IGV File Formats</a>, coveres all of the formats usable in the Broad Institute's Integrative Genomics Viewer software (which is a lot).</li>
  <li><a href="https://wiki.nci.nih.gov/display/TCGA/File+Format+Specifications" rel="nofollow noreferrer">NCI File Formats</a>, mostly formats used by TGCA (including MAF and VCF).</li>
  <li><a href="https://cgwb.nci.nih.gov/FAQ/FAQformat.html" rel="nofollow noreferrer">UCSC Genomics</a>, covers BED, MAF, and a few others.</li>
  <li><a href="http://software.broadinstitute.org/cancer/software/genepattern/file-formats-guide" rel="nofollow noreferrer">GenePattern</a>, covers many of the file formats related to microarray data.</li>
  <li><a href="http://software.broadinstitute.org/cancer/software/gsea/wiki/index.php/Data_formats" rel="nofollow noreferrer">GSEA</a>, Broad Gene Set Enrichment Analysis documentation.</li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-06-02T17:08:55.260Z
Answers.LastActivityDate: 2017-06-02T17:08:55.260Z
Answers.UserDisplayName: woemler
Answers.Body: <p>You mention Biopython, which contains tests: <a href="https://github.com/biopython/biopython/tree/master/Tests" rel="noreferrer">https://github.com/biopython/biopython/tree/master/Tests</a>.</p>
  
  <p>Some of the tests consist in reading files present in the folders listed in the above link. These files could be a starting point for a database of test files. Whenever one comes across a test case not covered with these files, one could construct a new test file and contribute it to Biopython, along with a test, or at least file an issue: <a href="https://github.com/biopython/biopython/issues" rel="noreferrer">https://github.com/biopython/biopython/issues</a>.</p>
  
  <p>That would be a way to contribute to Biopython while constituting a database of test files.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-06-02T17:00:27.620Z
Answers.Text: Thanks, I like this idea; I wonder if BioPerl and other projects also contain similar test suites that could be used
Answers.UserDisplayName: Chris_Rands
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-07-25T09:42:54.987Z
Answers.Text: A "me too" here: I've sometimes used the example or test data for BioPython as test data for other programs. You at least have the assurance of it being valid. You might find some similar useful data with the Galaxy tools.
Answers.UserDisplayName: agapow
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-06-02T16:18:48.537Z
Answers.LastActivityDate: 2017-06-02T16:18:48.537Z
Answers.UserDisplayName: bli
Answers.Body: <p>No. At least none that I've heard of and I doubt there ever will be. There is no central repository for formats and each tool, community, field etc have their own. </p>
  
  <p>The best you can do is look up the official standard for each format and hope they include examples. Having a truly comprehensive collection of all possible variations is basically impossible for <del>horrible</del> complex formats like VCF. Just consider the simple fact that it allows user-defined <code>INFO</code> fields with arbitrary contents. </p>
  
  <p>So the best you can do is make sure your scripts conform with the standard and hope your input does as well. </p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-06-02T14:51:45.370Z
Answers.Text: There could be a central repository though with templates for the most common formats. I don't see why not? It can't be completely comprehensive of course. To me this seems like a good idea...
Answers.UserDisplayName: Chris_Rands
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-06-02T14:56:10.577Z
Answers.Text: @Chris_Rands as a general rule, there are sadly very few centralized *anything* in biology and that's a general issue with the field. That said, I admit I don't really see the use for this. What you need is the standard. Multiple examples of the standard being applied in different ways doesn't seem all that helpful to me; at least not if the standard is well defined (which is not always the case). Come to think of it, I can't think of any such repository of format examples in general, not only in biology.
Answers.UserDisplayName: terdon
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-06-02T15:02:40.413Z
Answers.Text: Thanks, well I agree about the lack of standardisation/centerlisation. The main use for me would be test cases/unit testing. Knowing the specification is one thing, but spotting all the edge cases can be tricky without testing
Answers.UserDisplayName: Chris_Rands
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-06-02T15:11:16.897Z
Answers.Text: @Chris_Rands s/tricky/impossible :) I've now been working in industry after a decade and a half in academia and you wouldn't *believe* the crap we've been sent as "vcf files". There's no way to cover all the edge cases, some people will give you an excel file with a .vcf extension and expect it to work :(
Answers.UserDisplayName: terdon
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-10-16T12:55:38.633Z
Answers.Text: I think a github repo with examples of edge cases for each format is a great idea - everytime any of us came across a new way some one had done something werid with a format we could issue a PR on repo.
Answers.UserDisplayName: Ian Sudbery
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-06-02T14:24:55.750Z
Answers.LastActivityDate: 2017-06-02T14:24:55.750Z
Answers.UserDisplayName: terdon
Answers.Body: <p>These may not be exactly what your are looking for, but they do contain a wide range of formats with examples. If you want more, you might try searching other major databases hosting other types of into e.g. Uniprot, PDB, NCBI. </p>
  
  <p><a href="https://genome.ucsc.edu/FAQ/FAQformat.html" rel="nofollow noreferrer">https://genome.ucsc.edu/FAQ/FAQformat.html</a></p>
  
  <p><a href="http://www.ensembl.org/info/website/upload/bed.html" rel="nofollow noreferrer">http://www.ensembl.org/info/website/upload/bed.html</a></p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-06-02T14:32:03.510Z
Answers.Text: Thanks, I'm familiar with UCSC and Ensembl and their format descriptions but that's not really what I meant by templates
Answers.UserDisplayName: Chris_Rands
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-06-02T14:24:46.343Z
Answers.LastActivityDate: 2017-06-02T14:24:46.343Z
Answers.UserDisplayName: tweirick
Body: <p>I want some templates of different file formats that I can use to test my scripts and identify possible bugs in my code.</p>
  
  <p>For example, consider nucleotide FASTA, a simple but often abused format, I would want templates to capture regular and irregular formats, like I have seen all of these:</p>
  
  <p>1) Single line sequence</p>
  
  <pre><code>&gt;1
  ATG
  </code></pre>
  
  <p>2) Multi-line sequence</p>
  
  <pre><code>&gt;1
  AT
  G
  </code></pre>
  
  <p>3) Upper and lower case letters in sequence</p>
  
  <pre><code>&gt;1
  Atg
  </code></pre>
  
  <p>4) Ns and Xs (and possibly other letters) in sequence</p>
  
  <pre><code>&gt;1
  ANnxX
  </code></pre>
  
  <p>5) Unusual headers (sometimes non-ASCI characters, need to consider the encoding)</p>
  
  <pre><code>&gt;ATG &gt;汉字
  ATG
  </code></pre>
  
  <p>6) Whitespace between records</p>
  
  <pre><code>&gt;1
  ATG
  
  &gt;2
  ATG
  </code></pre>
  
  <p>7) Duplicated headers</p>
  
  <pre><code>&gt;1
  ATG
  &gt;1
  ATC
  </code></pre>
  
  <p>8) Empty headers or sequences (valid FASTA?)</p>
  
  <pre><code>&gt;
  &gt;
  </code></pre>
  
  <p>9) No new line <code>'\n'</code> character on last line (can mess-up file concatenation)</p>
  
  <pre><code>&gt;1
  A# &lt; no new-line here
  </code></pre>
  
  <p>10) Different newline characters depending on the OS</p>
  
  <pre><code>&gt;1
  A# \r\n vs \n
  </code></pre>
  
  <p>etc.</p>
  
  <p>There should be separate templates for nucleotide and protein FASTA, and separate ones for aligned FASTA.</p>
  
  <p>It would ideally include other aspects too, like different compression formats (such as <code>.gz</code>, <code>.bzip2</code>) and different file extensions (such as <code>.fa</code>, <code>.fasta</code>).</p>
  
  <p>I have never seen resources that provide templates covering these, but I think it would be useful. Of course I could build my own templates but it would take time to capture all the likely variations of the formats, particularly for more complex file formats.</p>
  
  <p>Note, I am not just interested in FASTA format, it was an example.</p>
  
  <p>Also note, I know about tools (such as <code>BioPython</code>) that should handle many formats well, but they may also have bugs. Anyway, in practice sometimes I end up parsing files myself directly because I don't want the overhead or dependency of an external package.</p>
  
  <p><em>EDIT: Please don't answer this question to say you don't know of any such resources, me neither, hence the question. <a href="https://bioinformatics.stackexchange.com/a/393/104">bli's helpful answer</a> shows that there is at least one test suite that could be used as a starting point. I know that it is normally easy to look up the specification of any particular file format.</em></p>
  
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-06-02T14:21:14.803Z
Comments.Text: Actually, the FASTA format definition is very simple. It only has two constraints and the second is often ignored: i) header lines must start with `>` and can contain anything they like except a `\n`. ii) sequence lines should be 60 characters per line. Nothing else is relevant, not the number of sequence lines, not the characters in the header, not the extension (which is usually irrelevant outside the Windows world anyway), there is no constraint on what characters a sequence can have (hence no difference between protein and nucleotide fasta specifications).
Comments.UserDisplayName: terdon
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-06-02T14:26:18.340Z
Comments.Text: Sure, the specification is simple, but as you say in practice it's not always followed and when you parse a file all these variations becomes relevant. I have had many tools trip up over strange FASTA headers or sequences and clearly it's important for the tools to distinguish nucleotides from amino acids. You may `glob` files with just 1 particular suffix for example.
Comments.UserDisplayName: Chris_Rands
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-06-02T14:33:54.623Z
Comments.Text: If the tools choke on that, it's the tools that are ignoring the standard. The FASTA standard is one of the very few that is actually clear and easy to understand. It is simply a very free format. And yes, of course you can glob on extensions. I, for example, tend to name my protein files `.pep` and nt `.fa` but there is no standard for this. All your tools need to do is i) take the entire line after a `>` as a sequence name and ii) everything else as the sequence. That's all, and that's all it take to be fasta.
Comments.UserDisplayName: terdon
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-06-02T14:39:48.213Z
Comments.Text: Sure, the tools may ignore the standard, and that's why I think it's good to have varied templates to test the tools. And it's not always simple, e.g. if you use FASTA headers to name subsequent files, special characters can be a nuisance. Anyway, as I said FASTA was just an example, the logic extends for other more complex formats
Comments.UserDisplayName: Chris_Rands
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-06-02T20:27:01.510Z
Comments.Text: You mention "I am not just interested in FASTA format, it was an example", but bear in mind that It's best to keep questions as specific as possible.
Comments.UserDisplayName: gringer
Tags: <fasta><database><file-formats>
Title: Are there any databases of templates for common bioinformatic file formats?
ThreadText: # Are there any databases of templates for common bioinformatic file formats?
  
  (Asked by: username_0 on 2017-06-02)
  
  I want some templates of different file formats that I can use to test my scripts and identify possible bugs in my code.
  
  For example, consider nucleotide FASTA, a simple but often abused format, I would want templates to capture regular and irregular formats, like I have seen all of these:
  
  1) Single line sequence
  
  ```
  >1
  ATG
  
  ```
  
  2) Multi-line sequence
  
  ```
  >1
  AT
  G
  
  ```
  
  3) Upper and lower case letters in sequence
  
  ```
  >1
  Atg
  
  ```
  
  4) Ns and Xs (and possibly other letters) in sequence
  
  ```
  >1
  ANnxX
  
  ```
  
  5) Unusual headers (sometimes non-ASCI characters, need to consider the encoding)
  
  ```
  >ATG >汉字
  ATG
  
  ```
  
  6) Whitespace between records
  
  ```
  >1
  ATG
  
  >2
  ATG
  
  ```
  
  7) Duplicated headers
  
  ```
  >1
  ATG
  >1
  ATC
  
  ```
  
  8) Empty headers or sequences (valid FASTA?)
  
  ```
  >
  >
  
  ```
  
  9) No new line `'\n'` character on last line (can mess-up file concatenation)
  
  ```
  >1
  A# < no new-line here
  
  ```
  
  10) Different newline characters depending on the OS
  
  ```
  >1
  A# \r\n vs \n
  
  ```
  
  etc.
  
  There should be separate templates for nucleotide and protein FASTA, and separate ones for aligned FASTA.
  
  It would ideally include other aspects too, like different compression formats (such as `.gz`, `.bzip2`) and different file extensions (such as `.fa`, `.fasta`).
  
  I have never seen resources that provide templates covering these, but I think it would be useful. Of course I could build my own templates but it would take time to capture all the likely variations of the formats, particularly for more complex file formats.
  
  Note, I am not just interested in FASTA format, it was an example.
  
  Also note, I know about tools (such as `BioPython`) that should handle many formats well, but they may also have bugs. Anyway, in practice sometimes I end up parsing files myself directly because I don't want the overhead or dependency of an external package.
  
  *EDIT: Please don't answer this question to say you don't know of any such resources, me neither, hence the question. [bli's helpful answer](https://bioinformatics.stackexchange.com/a/393/104) shows that there is at least one test suite that could be used as a starting point. I know that it is normally easy to look up the specification of any particular file format.*
  
  **Tags:** fasta, database, file-formats
  
  > **Comments:**
  > **username_1**: Actually, the FASTA format definition is very simple. It only has two constraints and the second is often ignored: i) header lines must start with `>` and can contain anything they like except a `\n`. ii) sequence lines should be 60 characters per line. Nothing else is relevant, not the number of sequence lines, not the characters in the header, not the extension (which is usually irrelevant outside the Windows world anyway), there is no constraint on what characters a sequence can have (hence no difference between protein and nucleotide fasta specifications).  
  > **username_0**: Sure, the specification is simple, but as you say in practice it's not always followed and when you parse a file all these variations becomes relevant. I have had many tools trip up over strange FASTA headers or sequences and clearly it's important for the tools to distinguish nucleotides from amino acids. You may `glob` files with just 1 particular suffix for example.  
  > **username_1**: If the tools choke on that, it's the tools that are ignoring the standard. The FASTA standard is one of the very few that is actually clear and easy to understand. It is simply a very free format. And yes, of course you can glob on extensions. I, for example, tend to name my protein files `.pep` and nt `.fa` but there is no standard for this. All your tools need to do is i) take the entire line after a `>` as a sequence name and ii) everything else as the sequence. That's all, and that's all it take to be fasta.  
  > **username_0**: Sure, the tools may ignore the standard, and that's why I think it's good to have varied templates to test the tools. And it's not always simple, e.g. if you use FASTA headers to name subsequent files, special characters can be a nuisance. Anyway, as I said FASTA was just an example, the logic extends for other more complex formats  
  > **username_2**: You mention "I am not just interested in FASTA format, it was an example", but bear in mind that It's best to keep questions as specific as possible.  
  
  ---
  
  ## 6 answers
  
  ### Answer by username_3 on 2020-03-31. Score: 2 | Accepted answer
  
  BioJulia maintains [BioFmtSpecimens](https://github.com/BioJulia/BioFmtSpecimens). From the README:
  
  > Bioinformatics is rife with formats and parsers for those formats. These parsers don't always agree on the definitions of these formats, since many lack any sort of formal standard.
  >
  > This repository aims to consolidate a collection of format specimens to create a unified set of data with which to test software against. Testing against the same cases is a first step towards agreeing on the details and edge cases of a format.
  
  
  ---
  
  ### Answer by username_6 on 2017-06-02. Score: 7
  
  You mention Biopython, which contains tests: <https://github.com/biopython/biopython/tree/master/Tests>.
  
  Some of the tests consist in reading files present in the folders listed in the above link. These files could be a starting point for a database of test files. Whenever one comes across a test case not covered with these files, one could construct a new test file and contribute it to Biopython, along with a test, or at least file an issue: <https://github.com/biopython/biopython/issues>.
  
  That would be a way to contribute to Biopython while constituting a database of test files.
  
  > **Comments:**
  > **username_0**: Thanks, I like this idea; I wonder if BioPerl and other projects also contain similar test suites that could be used  
  > **username_7**: A "me too" here: I've sometimes used the example or test data for BioPython as test data for other programs. You at least have the assurance of it being valid. You might find some similar useful data with the Galaxy tools.  
  
  ---
  
  ### Answer by username_4 on 2017-06-02. Score: 5
  
  Not that I am aware. It is best to go with format specifications when coding.
  
  Also it may be good to look at the example files that come together with various tools performing file conversions and handling. E.g.
  
  * Trimmomatic comes with few .fa files (<http://www.usadellab.org/cms/?page=trimmomatic>)
  * Samtools with a toy.sam (<https://github.com/lh3/samtools-legacy/blob/master/examples/toy.sam>)
  
  
  ---
  
  ### Answer by username_5 on 2017-06-02. Score: 3
  
  As far as I know, there is no single repository that collects all of the common data formats used in bioinformatics. Typically, you have to go to the source to find the specifications for each format. There are a few places that collect descriptions of file formats, though:
  
  * [IGV File Formats](https://software.broadinstitute.org/software/igv/FileFormats), coveres all of the formats usable in the Broad Institute's Integrative Genomics Viewer software (which is a lot).
  * [NCI File Formats](https://wiki.nci.nih.gov/display/TCGA/File+Format+Specifications), mostly formats used by TGCA (including MAF and VCF).
  * [UCSC Genomics](https://cgwb.nci.nih.gov/FAQ/FAQformat.html), covers BED, MAF, and a few others.
  * [GenePattern](http://software.broadinstitute.org/cancer/software/genepattern/file-formats-guide), covers many of the file formats related to microarray data.
  * [GSEA](http://software.broadinstitute.org/cancer/software/gsea/wiki/index.php/Data_formats), Broad Gene Set Enrichment Analysis documentation.
  
  
  ---
  
  ### Answer by username_1 on 2017-06-02. Score: 1
  
  No. At least none that I've heard of and I doubt there ever will be. There is no central repository for formats and each tool, community, field etc have their own.
  
  The best you can do is look up the official standard for each format and hope they include examples. Having a truly comprehensive collection of all possible variations is basically impossible for ~~horrible~~ complex formats like VCF. Just consider the simple fact that it allows user-defined `INFO` fields with arbitrary contents.
  
  So the best you can do is make sure your scripts conform with the standard and hope your input does as well.
  
  > **Comments:**
  > **username_0**: There could be a central repository though with templates for the most common formats. I don't see why not? It can't be completely comprehensive of course. To me this seems like a good idea...  
  > **username_1**: @username_0 as a general rule, there are sadly very few centralized *anything* in biology and that's a general issue with the field. That said, I admit I don't really see the use for this. What you need is the standard. Multiple examples of the standard being applied in different ways doesn't seem all that helpful to me; at least not if the standard is well defined (which is not always the case). Come to think of it, I can't think of any such repository of format examples in general, not only in biology.  
  > **username_0**: Thanks, well I agree about the lack of standardisation/centerlisation. The main use for me would be test cases/unit testing. Knowing the specification is one thing, but spotting all the edge cases can be tricky without testing  
  > **username_1**: @username_0 s/tricky/impossible :) I've now been working in industry after a decade and a half in academia and you wouldn't *believe* the crap we've been sent as "vcf files". There's no way to cover all the edge cases, some people will give you an excel file with a .vcf extension and expect it to work :(  
  > **username_8**: I think a github repo with examples of edge cases for each format is a great idea - everytime any of us came across a new way some one had done something werid with a format we could issue a PR on repo.  
  
  ---
  
  ### Answer by username_9 on 2017-06-02. Score: 0
  
  These may not be exactly what your are looking for, but they do contain a wide range of formats with examples. If you want more, you might try searching other major databases hosting other types of into e.g. Uniprot, PDB, NCBI.
  
  <https://genome.ucsc.edu/FAQ/FAQformat.html>
  
  <http://www.ensembl.org/info/website/upload/bed.html>
  
  > **Comments:**
  > **username_0**: Thanks, I'm familiar with UCSC and Ensembl and their format descriptions but that's not really what I meant by templates  
  
  ---
  
  