Answers.Body: <p>FAST5 is a proprietary format developed by Oxford Nanopore Technologies, and there's not much great documentation online. All FAST5 files will have the <code>Raw/</code> field, which contains the original measured <code>Signal</code>. Additional <code>Analyses/</code> fields can be added by tools such as basecallers which convert signal-&gt;ACGT (<a href="https://denbi-nanopore-training-course.readthedocs.io/en/latest/basecalling/basecalling.html" rel="nofollow noreferrer">Guppy</a>) or signal-level aligners (<a href="https://github.com/nanoporetech/tombo" rel="nofollow noreferrer">Tombo</a>).</p>
  <p>In particular, <code>/Analyses/Basecall_2D_000/</code> contains the results of 2D basecalling, added by Guppy when run with the <code>--post-out</code> flag. I think Tombo adds the event-level annotations you mention (k-mer mean/std/length). In 2D basecalling, both the physical forward and reverse strands for the same stretch of DNA are measured and basecalled. The basecalls will contain errors, so these will not be exact reverse complements of one another.</p>
  <p>Hope this answers your questions, I have a slightly more in-depth explanation on my blog: <a href="https://timd.one/blog/genomics/file_formats.php" rel="nofollow noreferrer">https://timd.one/blog/genomics/file_formats.php</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-19T21:09:47.973Z
Answers.Text: FWIW, this particular FAST5 was generated in 2017, before Guppy was available
Answers.UserDisplayName: gringer
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-19T14:41:44.340Z
Answers.LastActivityDate: 2022-02-19T14:41:44.340Z
Answers.UserDisplayName: TimD1
Answers.Body: <p>I'll give a different perspective. Oxford Nanopore FAST5 is NGS within a <code>HDF5</code> database. A clear description is <a href="https://bioinformatics.cvr.ac.uk/exploring-the-fast5-format/" rel="nofollow noreferrer">here</a>.</p>
  <p>HDF5 is a format used for rapid I/O of large data sets.</p>
  <p>Oxford Nano appear to have written a bit boilerplate around HDF5 and called it something that sounds like 'fasta'.</p>
  <p>A cool part of HDF5 (not FAST5) is  <code>HDF5 View</code>, which can be downloaded <a href="https://www.hdfgroup.org/downloads/hdfview/" rel="nofollow noreferrer">here</a>. Its cool because you can visualise the file structure of HDF5 db via a GUI. I very much suspect if you use HDF5 View on your FAST5 database you will be able to navigate (see not change) the entire file structure.</p>
  <p>To answer your question directly <code>h5ls</code> is listing the HDF5 database. To construct a HDF5 db you assign the data under a series of 'keys' which forms the directory structure. However, you perhaps might not need Oxford tools, because  Python pandas have direct series of methods to connect, construct and manipulate HDF5.</p>
  <p>I raised the HDF5 in a question then answered here on Bioinfo SE <a href="https://bioinformatics.stackexchange.com/questions/8724/hdf5-and-biosql-solutions">here</a>.</p>
  <p>Again the reason for HDF5's importance is unlike an SQL connection, data transfer and disconnect, HDF5 shifts data very quickly.</p>
  <p>I do question how Oxford Nanopore can lay claim to FAST5 because HDF5 is open source and always has been. If they want to lay claim to some boilerplate, be my guest - I write the stuff all the time ;-)</p>
  <p>HDF5 however is a serious project with major development time and appears to still be under active development.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-19T16:26:24.143Z
Answers.LastActivityDate: 2022-02-19T16:51:13.640Z
Answers.LastEditDate: 2022-02-19T16:51:13.640Z
Answers.UserDisplayName: M__
Body: <p>I would like to reproduce the results on figure 1 from <a href="https://www.nature.com/articles/nmeth.4184" rel="nofollow noreferrer">THIS</a> publication.
  I downloaded the data and explore it with h5ls and h5dump <a href="https://bioinformatics.cvr.ac.uk/exploring-the-fast5-format/" rel="nofollow noreferrer">based on HDF5</a>.
  I would like to understand:</p>
  <ol>
  <li><code>h5ls -r &lt;_fast5file_&gt; | grep Dataset</code>:</li>
  </ol>
  <pre><code>/Analyses/Basecall_2D_000/BaseCalled_2D/Alignment Dataset {6925}
  /Analyses/Basecall_2D_000/BaseCalled_2D/Fastq Dataset {SCALAR}
  /Analyses/Basecall_2D_000/BaseCalled_complement/Events Dataset {5576}
  /Analyses/Basecall_2D_000/BaseCalled_complement/Fastq Dataset {SCALAR}
  /Analyses/Basecall_2D_000/BaseCalled_complement/Model Dataset {4096}
  /Analyses/Basecall_2D_000/BaseCalled_template/Events Dataset {5793}
  /Analyses/Basecall_2D_000/BaseCalled_template/Fastq Dataset {SCALAR}
  /Analyses/Basecall_2D_000/BaseCalled_template/Model Dataset {4096}
  /Analyses/Basecall_2D_000/HairpinAlign/Alignment Dataset {5107}
  /Analyses/Basecall_2D_000/Log Dataset {SCALAR}
  /Analyses/Calibration_Strand_000/Log Dataset {SCALAR}
  /Analyses/EventDetection_000/Reads/Read_0/Events Dataset {11465}
  </code></pre>
  <p>Alignment? Then <code>complement/Fastq</code> is not the complement of <code>template/Fastq</code> (analyzed with h5dump). What is it? Also, events have sometimes 4</p>
  <ol start="2">
  <li>Some other datasets that were acquired somewhere else have the raw data and fastq but there is no associated information about kmer and mean(current)/std/length/etc... How do I generate events from raw data?</li>
  </ol>
  <p>I went through all documentations and tutorials that I could found in the web but couldn't find anything simple and complete. It feels as if it was a secret.</p>
  <p>Example <em>fast5file</em> would be <code>ecoli_er2925.MSssI.timp.100215.fast5/pass/kelvin_100215_ecolisssI_2515_1_ch195_file0_strand.fast5</code> from <a href="http://www.ebi.ac.uk/ena/data/view/PRJEB13021" rel="nofollow noreferrer">PRJEB13021</a></p>
  
Tags: <nanopore><fast5><hdf5>
Title: Understanding fast5 format
ThreadText: # Understanding fast5 format
  
  (Asked by: username_0 on 2022-02-19)
  
  I would like to reproduce the results on figure 1 from [THIS](https://www.nature.com/articles/nmeth.4184) publication.
  I downloaded the data and explore it with h5ls and h5dump [based on HDF5](https://bioinformatics.cvr.ac.uk/exploring-the-fast5-format/).
  I would like to understand:
  
  1. `h5ls -r <_fast5file_> | grep Dataset`:
  
  ```
  /Analyses/Basecall_2D_000/BaseCalled_2D/Alignment Dataset {6925}
  /Analyses/Basecall_2D_000/BaseCalled_2D/Fastq Dataset {SCALAR}
  /Analyses/Basecall_2D_000/BaseCalled_complement/Events Dataset {5576}
  /Analyses/Basecall_2D_000/BaseCalled_complement/Fastq Dataset {SCALAR}
  /Analyses/Basecall_2D_000/BaseCalled_complement/Model Dataset {4096}
  /Analyses/Basecall_2D_000/BaseCalled_template/Events Dataset {5793}
  /Analyses/Basecall_2D_000/BaseCalled_template/Fastq Dataset {SCALAR}
  /Analyses/Basecall_2D_000/BaseCalled_template/Model Dataset {4096}
  /Analyses/Basecall_2D_000/HairpinAlign/Alignment Dataset {5107}
  /Analyses/Basecall_2D_000/Log Dataset {SCALAR}
  /Analyses/Calibration_Strand_000/Log Dataset {SCALAR}
  /Analyses/EventDetection_000/Reads/Read_0/Events Dataset {11465}
  
  ```
  
  Alignment? Then `complement/Fastq` is not the complement of `template/Fastq` (analyzed with h5dump). What is it? Also, events have sometimes 4
  
  2. Some other datasets that were acquired somewhere else have the raw data and fastq but there is no associated information about kmer and mean(current)/std/length/etc... How do I generate events from raw data?
  
  I went through all documentations and tutorials that I could found in the web but couldn't find anything simple and complete. It feels as if it was a secret.
  
  Example *fast5file* would be `ecoli_er2925.MSssI.timp.100215.fast5/pass/kelvin_100215_ecolisssI_2515_1_ch195_file0_strand.fast5` from [PRJEB13021](http://www.ebi.ac.uk/ena/data/view/PRJEB13021)
  
  **Tags:** nanopore, fast5, hdf5
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2022-02-19. Score: 2
  
  FAST5 is a proprietary format developed by Oxford Nanopore Technologies, and there's not much great documentation online. All FAST5 files will have the `Raw/` field, which contains the original measured `Signal`. Additional `Analyses/` fields can be added by tools such as basecallers which convert signal->ACGT ([Guppy](https://denbi-nanopore-training-course.readthedocs.io/en/latest/basecalling/basecalling.html)) or signal-level aligners ([Tombo](https://github.com/nanoporetech/tombo)).
  
  In particular, `/Analyses/Basecall_2D_000/` contains the results of 2D basecalling, added by Guppy when run with the `--post-out` flag. I think Tombo adds the event-level annotations you mention (k-mer mean/std/length). In 2D basecalling, both the physical forward and reverse strands for the same stretch of DNA are measured and basecalled. The basecalls will contain errors, so these will not be exact reverse complements of one another.
  
  Hope this answers your questions, I have a slightly more in-depth explanation on my blog: <https://timd.one/blog/genomics/file_formats.php>
  
  > **Comments:**
  > **username_2**: FWIW, this particular FAST5 was generated in 2017, before Guppy was available  
  
  ---
  
  ### Answer by username_3 on 2022-02-19. Score: 1
  
  I'll give a different perspective. Oxford Nanopore FAST5 is NGS within a `HDF5` database. A clear description is [here](https://bioinformatics.cvr.ac.uk/exploring-the-fast5-format/).
  
  HDF5 is a format used for rapid I/O of large data sets.
  
  Oxford Nano appear to have written a bit boilerplate around HDF5 and called it something that sounds like 'fasta'.
  
  A cool part of HDF5 (not FAST5) is `HDF5 View`, which can be downloaded [here](https://www.hdfgroup.org/downloads/hdfview/). Its cool because you can visualise the file structure of HDF5 db via a GUI. I very much suspect if you use HDF5 View on your FAST5 database you will be able to navigate (see not change) the entire file structure.
  
  To answer your question directly `h5ls` is listing the HDF5 database. To construct a HDF5 db you assign the data under a series of 'keys' which forms the directory structure. However, you perhaps might not need Oxford tools, because Python pandas have direct series of methods to connect, construct and manipulate HDF5.
  
  I raised the HDF5 in a question then answered here on Bioinfo SE [here](https://bioinformatics.stackexchange.com/questions/8724/hdf5-and-biosql-solutions).
  
  Again the reason for HDF5's importance is unlike an SQL connection, data transfer and disconnect, HDF5 shifts data very quickly.
  
  I do question how Oxford Nanopore can lay claim to FAST5 because HDF5 is open source and always has been. If they want to lay claim to some boilerplate, be my guest - I write the stuff all the time ;-)
  
  HDF5 however is a serious project with major development time and appears to still be under active development.
  
  
  ---
  
  