Answers.Body: <p>It looks like you're creating too much additional work for yourself. What I'm trying to lead to with my comments is that the file names should help in working out if a file represents single reads or multiple reads, in which case the single reads can be shifted into a separate folder.</p>
  <p>Based on the file names that I would expect, a search for files containing '_read_[0-9]' should be sufficient to fish out the single fast5 files:</p>
  <pre><code>find . -name '*_read_[0-9]*' &gt; single_reads.txt
  mkdir -p single_reads
  cat single_reads.txt | while read fname; do mv -i ${fname} single_reads; done
  </code></pre>
  <p><em>[That <code>single_reads.txt</code> file could be split into multiple files if directory entries get too large.]</em></p>
  <p>After which you should have a single folder that contains thousands of single-read fast5 files, and other folders that contain multi-fast5 files, removing the problem of needing to deal with mixed file types.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-12-13T21:08:00.113Z
Answers.Text: I am gonna try this, thank you! @gringer
Answers.UserDisplayName: Marjan
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-12-10T05:36:26.217Z
Answers.LastActivityDate: 2023-12-14T02:44:44.233Z
Answers.LastEditDate: 2023-12-14T02:44:44.233Z
Answers.UserDisplayName: gringer
Body: <p>I have a dataset that contains thousands of mixed multiple and single fast5 files in a non-homogenous folder structure.
  The reads and fast5 files are coming from different labs and some labs gave single fast5 files and some multi fast5 files, without clear mappings. My final goal is to convert all the fast5 files to multi fast5 files.</p>
  <p>There are commands in ont_fast5_api which is an interface to HDF5 files of the Oxford Nanopore.
  Using this API we can convert single fast5 files and multi fast5 files to each other.
  this is the link to this API github page:
  <a href="https://github.com/nanoporetech/ont_fast5_api" rel="nofollow noreferrer">https://github.com/nanoporetech/ont_fast5_api</a></p>
  <p>So I made a test mixed folder to check if I can use the API without missing reads as a result of conversion. The test folder structure is similar to my original folder structure:</p>
  <ul>
  <li>mixed
  <ul>
  <li>Subfolder_A
  <ul>
  <li>single_fast5 x 4000</li>
  </ul>
  </li>
  <li>Subfolder_B
  <ul>
  <li>multiple_fast5 (contains 4000 single reads fast5)</li>
  </ul>
  </li>
  <li>single_fast5 x 2</li>
  <li>multiple_fast5_1 (contains 4000 single reads fast5)</li>
  <li>multiple_fast5_2 (contains 4000 single reads fast5)</li>
  <li>multiple_fast5_3 (contains 100 single reads fast5)</li>
  </ul>
  </li>
  </ul>
  <p>Here is a screenshot of the structure with a few files from each folder:</p>
  <p><img src="https://cdn.discordapp.com/attachments/991841204088098977/1183239459714646037/mixed_tree.png?ex=65879c9e&amp;is=6575279e&amp;hm=0747962eff4b5167c6026c1d7a9612d66983bcab5fb3ce5a3315166b2c19dc5d&amp;" alt="alt text" title="folder structure" /></p>
  <p>and I created two other folders outside of the mixed folder as follows:</p>
  <ul>
  <li><p>single</p>
  </li>
  <li><p>multi</p>
  </li>
  </ul>
  <p>Here my question is about <em>single_to_multi_fast5</em> and <em>multi_to_single_fast5</em> commands.</p>
  <pre><code>test_path=mixed
  multi_path=multi
  single_path=single
  multi_to_single_fast5 -i $test_path/ -s $single_path/ --recursive
  </code></pre>
  <p>in the $single_path, I expect to see, all of the reads in multi files converted to single files (12100 reads in 12100 single fast5 files), but I see 8100 single files.</p>
  <p><img src="https://cdn.discordapp.com/attachments/991841204088098977/1183088499252023406/single.png?ex=65871006&amp;is=65749b06&amp;hm=484c575feb7ddb4effbb6f05bbb91fe64290c7da4d1c25b4339aaffcccd77d50&amp;" alt="alt text" title="single path" />
  So I have 4000 missing reads.</p>
  <p>Now I convert single read fast5 file to multi:</p>
  <pre><code>single_to_multi_fast5 -i $test_path/ -s $multi_path/ --filename_base $output_name --batch_size 1000 --recursive
  </code></pre>
  <p>in the $multi_path folder, I expect to see that generated multi files contain all my single-read files (4000 + 2), but I see 3006 in the mapping summary.</p>
  <p><img src="https://cdn.discordapp.com/attachments/991841204088098977/1183089993523154984/multi.png?ex=6587116b&amp;is=65749c6b&amp;hm=6d78f50b7a5ede6581c7b0205d1db5e74d88397963907ebed457ce6f8f487592&amp;" alt="alt text" title="mutli path" /></p>
  <p>So again I missed 996 single reads in my multi files.
  Since I made a small subsample of data, I am sure that the files are not overwritten.</p>
  <p>Because my final goal is <strong>converting everything to multi</strong> I was going to do the following <strong>pipeline</strong>:</p>
  <pre><code>orig_path=original_folder
  intermediate_path=intermediate
  final_path=final_multi
  # converts/collects all multi files in the original folder
  multi_to_single_fast5 -i <span class="math-container">$orig_path/ -s $</span>intermediate_path/ --recursive
  single_to_multi_fast5 -i <span class="math-container">$intermediate_path/ -s $</span>final_path/ --filename_base $output_name --batch_size 1000 --recursive
  
  # converts/collects all single files in the original folder
  single_to_multi_fast5 -i <span class="math-container">$orig_path/ -s $</span>final_path/ --filename_base $output_name --batch_size 1000 --recursive
  
  </code></pre>
  <p>However because it seems I get missing reads during the process, I cannot use this pipeline.</p>
  <p>At the same time, my original dataset is super huge, I cannot check if individual files are single or multi. It would take ages (I guess).
  Is there a solution for solving this problem besides checking every file? Maybe my pipeline is not quite suitable for what I want to do.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-12-09T20:20:32.723Z
Comments.Text: Can you please show the first few filenames in each folder? e.g. by running the following command: `find mixed -type 'd' | while read dname; do echo "** ${dname} **"; tree -L 1 ${dname} | head; done`
Comments.UserDisplayName: gringer
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-12-10T02:56:09.893Z
Comments.Text: Thank you for your reply! I included the first few filenames in each folder @gringer
Comments.UserDisplayName: Marjan
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-12-10T05:25:28.453Z
Comments.Text: The image seems like it's annotated incorrectly. A file with a channel and read number should be a single fast5 file, not a multi fast5 file.
Comments.UserDisplayName: gringer
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-12-13T21:06:12.563Z
Comments.Text: Yes, the 2nd label (multi fast5 files with 4000 reads) and the 3rd label (single fast5 files) should be swapped. @gringer
Comments.UserDisplayName: Marjan
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-12-14T02:44:01.150Z
Comments.Text: Okay, that's good to know; thanks for the clarification.
Comments.UserDisplayName: gringer
Tags: <nanopore><fast5>
Title: single_to_multi_fast5 does not collect all the single files if the input folder contains mixed types of fast5 files
ThreadText: # single_to_multi_fast5 does not collect all the single files if the input folder contains mixed types of fast5 files
  
  (Asked by: username_0 on 2023-12-03)
  
  I have a dataset that contains thousands of mixed multiple and single fast5 files in a non-homogenous folder structure.
  The reads and fast5 files are coming from different labs and some labs gave single fast5 files and some multi fast5 files, without clear mappings. My final goal is to convert all the fast5 files to multi fast5 files.
  
  There are commands in ont\_fast5\_api which is an interface to HDF5 files of the Oxford Nanopore.
  Using this API we can convert single fast5 files and multi fast5 files to each other.
  this is the link to this API github page:
  <https://github.com/nanoporetech/ont_fast5_api>
  
  So I made a test mixed folder to check if I can use the API without missing reads as a result of conversion. The test folder structure is similar to my original folder structure:
  
  * mixed
    + Subfolder\_A
      - single\_fast5 x 4000
    + Subfolder\_B
      - multiple\_fast5 (contains 4000 single reads fast5)
    + single\_fast5 x 2
    + multiple\_fast5\_1 (contains 4000 single reads fast5)
    + multiple\_fast5\_2 (contains 4000 single reads fast5)
    + multiple\_fast5\_3 (contains 100 single reads fast5)
  
  Here is a screenshot of the structure with a few files from each folder:
  
  ![alt text](https://cdn.discordapp.com/attachments/991841204088098977/1183239459714646037/mixed_tree.png?ex=65879c9e&is=6575279e&hm=0747962eff4b5167c6026c1d7a9612d66983bcab5fb3ce5a3315166b2c19dc5d& "folder structure")
  
  and I created two other folders outside of the mixed folder as follows:
  
  * single
  * multi
  
  Here my question is about *single\_to\_multi\_fast5* and *multi\_to\_single\_fast5* commands.
  
  ```
  test_path=mixed
  multi_path=multi
  single_path=single
  multi_to_single_fast5 -i $test_path/ -s $single_path/ --recursive
  
  ```
  
  in the $single\_path, I expect to see, all of the reads in multi files converted to single files (12100 reads in 12100 single fast5 files), but I see 8100 single files.
  
  ![alt text](https://cdn.discordapp.com/attachments/991841204088098977/1183088499252023406/single.png?ex=65871006&is=65749b06&hm=484c575feb7ddb4effbb6f05bbb91fe64290c7da4d1c25b4339aaffcccd77d50& "single path")
  So I have 4000 missing reads.
  
  Now I convert single read fast5 file to multi:
  
  ```
  single_to_multi_fast5 -i $test_path/ -s $multi_path/ --filename_base $output_name --batch_size 1000 --recursive
  
  ```
  
  in the $multi\_path folder, I expect to see that generated multi files contain all my single-read files (4000 + 2), but I see 3006 in the mapping summary.
  
  ![alt text](https://cdn.discordapp.com/attachments/991841204088098977/1183089993523154984/multi.png?ex=6587116b&is=65749c6b&hm=6d78f50b7a5ede6581c7b0205d1db5e74d88397963907ebed457ce6f8f487592& "mutli path")
  
  So again I missed 996 single reads in my multi files.
  Since I made a small subsample of data, I am sure that the files are not overwritten.
  
  Because my final goal is **converting everything to multi** I was going to do the following **pipeline**:
  
  ```
  orig_path=original_folder
  intermediate_path=intermediate
  final_path=final_multi
  # converts/collects all multi files in the original folder
  multi_to_single_fast5 -i $orig_path/ -s $intermediate_path/ --recursive
  single_to_multi_fast5 -i $intermediate_path/ -s $final_path/ --filename_base $output_name --batch_size 1000 --recursive
  
  # converts/collects all single files in the original folder
  single_to_multi_fast5 -i $orig_path/ -s $final_path/ --filename_base $output_name --batch_size 1000 --recursive
  
  ```
  
  However because it seems I get missing reads during the process, I cannot use this pipeline.
  
  At the same time, my original dataset is super huge, I cannot check if individual files are single or multi. It would take ages (I guess).
  Is there a solution for solving this problem besides checking every file? Maybe my pipeline is not quite suitable for what I want to do.
  
  **Tags:** nanopore, fast5
  
  > **Comments:**
  > **username_1**: Can you please show the first few filenames in each folder? e.g. by running the following command: `find mixed -type 'd' | while read dname; do echo "** ${dname} **"; tree -L 1 ${dname} | head; done`  
  > **username_0**: Thank you for your reply! I included the first few filenames in each folder @username_1  
  > **username_1**: The image seems like it's annotated incorrectly. A file with a channel and read number should be a single fast5 file, not a multi fast5 file.  
  > **username_0**: Yes, the 2nd label (multi fast5 files with 4000 reads) and the 3rd label (single fast5 files) should be swapped. @username_1  
  > **username_1**: Okay, that's good to know; thanks for the clarification.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-12-10. Score: 1 | Accepted answer
  
  It looks like you're creating too much additional work for yourself. What I'm trying to lead to with my comments is that the file names should help in working out if a file represents single reads or multiple reads, in which case the single reads can be shifted into a separate folder.
  
  Based on the file names that I would expect, a search for files containing '\_read\_[0-9]' should be sufficient to fish out the single fast5 files:
  
  ```
  find . -name '*_read_[0-9]*' > single_reads.txt
  mkdir -p single_reads
  cat single_reads.txt | while read fname; do mv -i ${fname} single_reads; done
  
  ```
  
  *[That `single_reads.txt` file could be split into multiple files if directory entries get too large.]*
  
  After which you should have a single folder that contains thousands of single-read fast5 files, and other folders that contain multi-fast5 files, removing the problem of needing to deal with mixed file types.
  
  > **Comments:**
  > **username_0**: I am gonna try this, thank you! @username_1  
  
  ---
  
  