Answers.Body: <p>I do this very frequently, using the read file name to identify FAST5 files associated with particular reads. If a FASTQ record includes the channel number and read number (and preferably the runID as well), then I use that information to find the associated FAST5 read.</p>
  
  <p>If reads are called with Albacore, then the <code>sequencing_summary.txt</code> file has additional information. The first column is the exact name of the file that was called, the second column is the read ID of the FASTQ sequence, and the third column is the run ID associated with that read ID. This is much easier to work with, but does require calling reads with Albacore (which, admittedly, is what seems to be producing the best results at the moment).</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-08-16T05:38:59.320Z
Answers.Text: yes this is what I did, basically: 1. extract fast5 to fastq; 2. align the fastq to the reference; 3. get the reads ID that map from the SAM of alignment; 4. grep the mapping reads ID from the fastq; 5. extract the name of the fast5 from the fasta header of each read (last field); 6. find the corresponding fast5 in the original fast5 directory. But I was wondering if I did not miss any software that could do this on one run.
Answers.UserDisplayName: aechchiki
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-08-16T07:39:08.467Z
Answers.Text: Added a note about albacore, which has a direct link between FASTQ read IDs and FAST5 file names
Answers.UserDisplayName: gringer
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-08-16T08:25:50.843Z
Answers.Text: In light of your above comment @AminaEchchiki I have added in compatibility for BAM and SAM files too.
Answers.UserDisplayName: Michael Hall
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-08-15T18:44:00.690Z
Answers.LastActivityDate: 2017-08-16T07:38:17.043Z
Answers.LastEditDate: 2017-08-16T07:38:17.043Z
Answers.UserDisplayName: gringer
Answers.Body: <p><strong>NOTICE</strong>:
  I have altered my answer slightly from the original as I have turned the original script into a <code>pip</code> installable program (with tests) and have updated the links and code snippets accordingly. The essence of the answer is still exactly the same.</p>
  
  <hr>
  
  <p>This is something I have been meaning to get around to for a while, so thanks for the prompt.</p>
  
  <p>I have <a href="https://github.com/mbhall88/fast5seek" rel="nofollow noreferrer">created a python program called <code>fast5seek</code></a> to do what (I think) you're after.</p>
  
  <p>As you have mentioned this is for educational purposes I have added a tonne of comments to the code too so I think you shouldn't have any issues following it.</p>
  
  <p>The docs on the <a href="https://github.com/mbhall88/fast5seek" rel="nofollow noreferrer">GitHub repo</a> have all the info, but for those reading along at home</p>
  
  <pre><code>pip3 install fast5seek
  fast5seek -i /path/to/fast5s -r in.fastq in.bam in.sam -o out.txt
  </code></pre>
  
  <p>What it does is read in <code>&lt;in.fastq|in.bam|in.sam&gt;</code> and extract the read id from each header. It then goes through all the fast5 files under <code>/path/to/fast5s</code> and checks whether their read id is in the set of read ids from <code>&lt;in.fastq|in.bam|in.sam&gt;</code>. If it is, the path to the file is written to it's own line in <code>out.txt</code>.</p>
  
  <p>If no output (<code>-o</code>) is given, it will write the output to stdout.</p>
  
  <p>So if you wanted to pipe these paths into another program, you could do something like</p>
  
  <pre><code>mkdir subset_dir/
  fast5seek -i /path/to/fast5s/ -r in.fastq | xargs cp -t subset_dir/
  </code></pre>
  
  <p>The above example would copy the fast5 files that are found in your fastq/BAM/SAM to <code>subset_dir/</code>.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-08-16T11:40:03.693Z
Answers.Text: As a caution, the run id should be checked (if that information is available). There's a small possibility that two FASTQ files from different runs will have the same read id.
Answers.UserDisplayName: gringer
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-08-17T00:56:10.317Z
Answers.Text: I have now added this check for fastq files. Although, run_id is not contained within BAM/SAM files (at least not for the ones I have) so this check will only occur on fastq.
Answers.UserDisplayName: Michael Hall
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2017-08-16T04:30:09.763Z
Answers.LastActivityDate: 2018-11-02T12:22:56.053Z
Answers.LastEditDate: 2018-11-02T12:22:56.053Z
Answers.UserDisplayName: Michael Hall
Body: <p><code>fast5</code> is a variant of <a href="https://support.hdfgroup.org/HDF5/whatishdf5.html" rel="noreferrer"><code>HDF5</code></a> the native format in which raw data from Oxford Nanopore MinION are provided. You can easily extract the reads in fast5 format into a standard fastq format, using for example <a href="https://poretools.readthedocs.io/en/latest/" rel="noreferrer"><code>poretools</code></a>. </p>
  
  <p>Say I have aligned these reads in <code>fastq</code> format to an external reference genome, resulting in a <code>SAM</code> file. Say I have then taken a subset of the  <code>SAM</code> file, according to the <a href="https://samtools.github.io/hts-specs/SAMv1.pdf" rel="noreferrer">bitwise flag</a>, to include only the reads that map to the reference. With the read ID, I can then grep them out from the file containing the reads in <code>fastq</code> format, generating a subset file in <code>fastq</code> format containing only the IDs that have mapped to the reference.  </p>
  
  <p>Now my question is, can we subset reads from the <code>fast5</code> archive according to the list of mapping reads as taken from the file with reads in <code>fastq</code> format? This is for educational purposes, so that we have a smaller starting archive, and the <code>fast5</code> -> <code>fastq</code> extraction takes less cpu time. </p>
  
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-08-15T16:18:05.127Z
Comments.Text: This seems a nice question to play around if we could have sample data. Could you provide (or a link to) a fast5 and the list of mappings ?
Comments.UserDisplayName: llrs
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-08-15T18:09:46.967Z
Comments.Text: There are some sample reads from my TEDx talk last year [here](https://www.dropbox.com/sh/o7fz7s91865u6vw/AAAsIzqbmrylD6lHl-yOJggXa?dl=0), reads from runs I did during the AGTA conference last year [here](https://www.dropbox.com/sh/0iqbjk7drmpsro0/AACBehmjVL7iABf5U2NVi9Lra?dl=0), and reads from a chimeric read investigation [here](http://www.ebi.ac.uk/ena/data/view/PRJEB20601). The ONT FAST5 "standard" change so frequently that it's difficult to find something that represents what's currently available.
Comments.UserDisplayName: gringer
Tags: <nanopore><software-recommendation><fastq><format-conversion><fast5>
Title: How to convert fastq to fast5
ThreadText: # How to convert fastq to fast5
  
  (Asked by: username_0 on 2017-08-15)
  
  `fast5` is a variant of [`HDF5`](https://support.hdfgroup.org/HDF5/whatishdf5.html) the native format in which raw data from Oxford Nanopore MinION are provided. You can easily extract the reads in fast5 format into a standard fastq format, using for example [`poretools`](https://poretools.readthedocs.io/en/latest/).
  
  Say I have aligned these reads in `fastq` format to an external reference genome, resulting in a `SAM` file. Say I have then taken a subset of the `SAM` file, according to the [bitwise flag](https://samtools.github.io/hts-specs/SAMv1.pdf), to include only the reads that map to the reference. With the read ID, I can then grep them out from the file containing the reads in `fastq` format, generating a subset file in `fastq` format containing only the IDs that have mapped to the reference.
  
  Now my question is, can we subset reads from the `fast5` archive according to the list of mapping reads as taken from the file with reads in `fastq` format? This is for educational purposes, so that we have a smaller starting archive, and the `fast5` -> `fastq` extraction takes less cpu time.
  
  **Tags:** nanopore, software-recommendation, fastq, format-conversion, fast5
  
  > **Comments:**
  > **username_1**: This seems a nice question to play around if we could have sample data. Could you provide (or a link to) a fast5 and the list of mappings ?  
  > **username_2**: There are some sample reads from my TEDx talk last year [here](https://www.dropbox.com/sh/o7fz7s91865u6vw/AAAsIzqbmrylD6lHl-yOJggXa?dl=0), reads from runs I did during the AGTA conference last year [here](https://www.dropbox.com/sh/0iqbjk7drmpsro0/AACBehmjVL7iABf5U2NVi9Lra?dl=0), and reads from a chimeric read investigation [here](http://www.ebi.ac.uk/ena/data/view/PRJEB20601). The ONT FAST5 "standard" change so frequently that it's difficult to find something that represents what's currently available.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_3 on 2017-08-16. Score: 6 | Accepted answer
  
  **NOTICE**:
  I have altered my answer slightly from the original as I have turned the original script into a `pip` installable program (with tests) and have updated the links and code snippets accordingly. The essence of the answer is still exactly the same.
  
  ---
  
  This is something I have been meaning to get around to for a while, so thanks for the prompt.
  
  I have [created a python program called `fast5seek`](https://github.com/mbhall88/fast5seek) to do what (I think) you're after.
  
  As you have mentioned this is for educational purposes I have added a tonne of comments to the code too so I think you shouldn't have any issues following it.
  
  The docs on the [GitHub repo](https://github.com/mbhall88/fast5seek) have all the info, but for those reading along at home
  
  ```
  pip3 install fast5seek
  fast5seek -i /path/to/fast5s -r in.fastq in.bam in.sam -o out.txt
  
  ```
  
  What it does is read in `<in.fastq|in.bam|in.sam>` and extract the read id from each header. It then goes through all the fast5 files under `/path/to/fast5s` and checks whether their read id is in the set of read ids from `<in.fastq|in.bam|in.sam>`. If it is, the path to the file is written to it's own line in `out.txt`.
  
  If no output (`-o`) is given, it will write the output to stdout.
  
  So if you wanted to pipe these paths into another program, you could do something like
  
  ```
  mkdir subset_dir/
  fast5seek -i /path/to/fast5s/ -r in.fastq | xargs cp -t subset_dir/
  
  ```
  
  The above example would copy the fast5 files that are found in your fastq/BAM/SAM to `subset_dir/`.
  
  > **Comments:**
  > **username_2**: As a caution, the run id should be checked (if that information is available). There's a small possibility that two FASTQ files from different runs will have the same read id.  
  > **username_3**: I have now added this check for fastq files. Although, run_id is not contained within BAM/SAM files (at least not for the ones I have) so this check will only occur on fastq.  
  
  ---
  
  ### Answer by username_2 on 2017-08-15. Score: 3
  
  I do this very frequently, using the read file name to identify FAST5 files associated with particular reads. If a FASTQ record includes the channel number and read number (and preferably the runID as well), then I use that information to find the associated FAST5 read.
  
  If reads are called with Albacore, then the `sequencing_summary.txt` file has additional information. The first column is the exact name of the file that was called, the second column is the read ID of the FASTQ sequence, and the third column is the run ID associated with that read ID. This is much easier to work with, but does require calling reads with Albacore (which, admittedly, is what seems to be producing the best results at the moment).
  
  > **Comments:**
  > **username_0**: yes this is what I did, basically: 1. extract fast5 to fastq; 2. align the fastq to the reference; 3. get the reads ID that map from the SAM of alignment; 4. grep the mapping reads ID from the fastq; 5. extract the name of the fast5 from the fasta header of each read (last field); 6. find the corresponding fast5 in the original fast5 directory. But I was wondering if I did not miss any software that could do this on one run.  
  > **username_2**: Added a note about albacore, which has a direct link between FASTQ read IDs and FAST5 file names  
  > **username_3**: In light of your above comment @AminaEchchiki I have added in compatibility for BAM and SAM files too.  
  
  ---
  
  