Answers.Body: <p>You didn't show us how the original .h5ad object was generated, how you read it in, how you wrote it, and if you did anything else in-between besides filtering. <a href="https://anndata.readthedocs.io/en/latest/generated/anndata.AnnData.write.html" rel="nofollow noreferrer"><code>AnnData.write()</code></a> takes a <code>compression</code> arg that might explain the difference. As also mentioned in that link, ensuring you are writing a sparse matrix can make a substantial difference to file size (the matrix in <code>adata.X</code> is generally by far the largest part of the object)</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-20T06:28:02.827Z
Answers.Text: I was taking this dataset https://data.humancellatlas.org/explore/projects/85a9263b-0887-48ed-ab1a-ddfa773727b6/get-curl-command
Answers.UserDisplayName: PesKchan
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-20T06:32:12.760Z
Answers.Text: I have updated the question
Answers.UserDisplayName: PesKchan
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-22T20:58:16.173Z
Answers.Text: I have updated my code please have a look and for this "You didn't show us how the original .h5ad object was generated" please have a look https://data.humancellatlas.org/explore/projects/85a9263b-0887-48ed-ab1a-ddfa773727b6
Answers.UserDisplayName: PesKchan
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-19T20:20:01.800Z
Answers.LastActivityDate: 2023-05-19T20:20:01.800Z
Answers.UserDisplayName: Chris_Rands
Answers.Body: <p>Scanpy's <code>h5ad</code> is in fact hdf5 format. This format is used for rapid read/write (i/o) large amounts of data and thats its purpose. Its fairly easy to use from scratch.</p>
  <p>Its purpose is not to compress data. If you are concerned with the contents of <code>h5ad</code> why not just read them directly via the <code>hdf5</code> viewer. You can download it here. <a href="https://www.hdfgroup.org/downloads/hdfview/" rel="nofollow noreferrer">https://www.hdfgroup.org/downloads/hdfview/</a></p>
  <p>The file will then open up as a &quot;Explorer&quot; (Windows) or &quot;Finder&quot; (OSX) GUI file system and you can open the various files and see whats in them.</p>
  <p>My suspicion is that the original data is been written into the file. 'cause it can i/o rapidly and access precisely what it needs then file size could be a secondary consideration. Anyway you can peer in and find out by looking at the original file and the hdf5 save you've done.</p>
  <p>An example from the website is here</p>
  <p><a href="https://i.sstatic.net/zhtSf.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/zhtSf.png" alt="enter image description here" /></a></p>
  <p>Cool right?</p>
  <p>Personally, I think HDF5 is overrated and <code>feather</code> outperforms it for speed (i/o) - but thats just for dataframes. Its not much faster than pickling in case you are interested and pickling in addition does compression. parquet is the format that does rapid i/o but specialises in compression, again mostly for dataframes.</p>
  <p>HDF5 however does permit very pretty GUI interfaces.</p>
  <hr />
  <p>Also if you want to load up and look at various files, that just uses the Python <code>hdf5</code> module.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-18T02:33:53.890Z
Answers.LastActivityDate: 2023-05-18T03:09:54.613Z
Answers.LastEditDate: 2023-05-18T03:09:54.613Z
Answers.UserDisplayName: M__
Body: <p>I'm trying to do simple filter for single data that is stored in <code>h5ad</code> file format using this</p>
  <pre><code>import scanpy as sc
  
  sc.pp.filter_cells(adata, min_genes=500)
  sc.pp.filter_genes(adata, min_cells=10)
  print(adata.n_obs, adata.n_vars)
  </code></pre>
  <p>when I saved the filtered data and see the size of my original data compare it, the size of the filtered data is more, but the dimension etc are reduced due to filtering.</p>
  <p>So is it something wrong am I doing?</p>
  <p><strong>UPDATE</strong></p>
  <pre><code>adata = sc.read_h5ad('/Single_cell/cerebellar_development/508cf892-174c-45ab-b2dc-05f54f1ee7ed/aldinger20.processed.h5ad')
  adata
  
  AnnData object with n_obs × n_vars = 69174 × 49166
      obs: 'orig.ident', 'nCount_RNA', 'nFeature_RNA', 'sample_id', 'percent.mito', 'S.Score', 'G2M.Score', 'Phase', 'CC.Difference', 'nCount_SCT', 'nFeature_SCT', 'age', 'figure_clusters', 'sex', 'type', 'experiment', 'fig_cell_type'
      var: 'name'
      obsm: 'X_pca', 'X_umap', 'X_tsne'
  
  
  sc.pp.filter_cells(adata, min_genes=500)
  sc.pp.filter_genes(adata, min_cells=10)
  print(adata.n_obs, adata.n_vars)
  
  results_file = 'adata.h5ad'
  adata.write(results_file)
  </code></pre>
  <p><strong>FULL CODE</strong></p>
  <h1>updated version: h5ad filter, added layers to make the filtered file usable for seurat to read and make a correct seurat object</h1>
  <pre><code>#https://github.com/satijalab/seurat/issues/2485
  
  import scanpy as sc
  import numpy as np
  import pandas as pd
  import time
  from ipywidgets import interact, interactive, widgets
  from IPython.display import display, HTML
  
  def filter_and_save_h5ad(h5ad_file, min_genes, min_cells):
      # h5ad file read
      adata = sc.read_h5ad(h5ad_file)
  
      # cells and genes filter parameter from user
      sc.pp.filter_cells(adata, min_genes=min_genes)
      sc.pp.filter_genes(adata, min_cells=min_cells)
  
      # Add 'counts' layer
      adata.layers['counts'] = adata.X.copy()
  
      # add 'winsorization' layer
      adata.layers['winsorized'] = adata.X.copy()
  
  
      # filtered dataset save
      filtered_file = h5ad_file.replace('.h5ad', '_filtered.h5ad')
      adata.write(filtered_file)
  
      print(f&quot;Filtered dataset saved to {filtered_file}&quot;)
  
  # input widgets
  h5ad_file_widget = widgets.Text(description='File_add:')
  min_genes_widget = widgets.IntText(description='Min_Genes:')
  min_cells_widget = widgets.IntText(description='Min_Cells:')
  button = widgets.Button(description='Filter_Save')
  
  # timer widget
  time_widget = widgets.HTML(value=&quot;&lt;p&gt;Time passed away: 0 seconds&lt;/p&gt;&quot;)
  
  def on_button_clicked(click_me):
      h5ad_file = h5ad_file_widget.value.strip()
      min_genes = min_genes_widget.value
      min_cells = min_cells_widget.value
      
      # timer
      start_time = time.time()
      
      filter_and_save_h5ad(h5ad_file, min_genes, min_cells)
      
      # Calculate and update the elapsed time
      elapsed_time = time.time() - start_time
      time_widget.value = f&quot;&lt;p&gt;Time passed away: {elapsed_time} seconds&lt;/p&gt;&quot;
  
  button.on_click(on_button_clicked)
  
  # Display the widgets and button
  display(h5ad_file_widget, min_genes_widget, min_cells_widget, button, time_widget)
  </code></pre>
  
Tags: <scrnaseq><single-cell>
Title: h5ad file format filter
ThreadText: # h5ad file format filter
  
  (Asked by: username_0 on 2023-05-17)
  
  I'm trying to do simple filter for single data that is stored in `h5ad` file format using this
  
  ```
  import scanpy as sc
  
  sc.pp.filter_cells(adata, min_genes=500)
  sc.pp.filter_genes(adata, min_cells=10)
  print(adata.n_obs, adata.n_vars)
  
  ```
  
  when I saved the filtered data and see the size of my original data compare it, the size of the filtered data is more, but the dimension etc are reduced due to filtering.
  
  So is it something wrong am I doing?
  
  **UPDATE**
  
  ```
  adata = sc.read_h5ad('/Single_cell/cerebellar_development/508cf892-174c-45ab-b2dc-05f54f1ee7ed/aldinger20.processed.h5ad')
  adata
  
  AnnData object with n_obs × n_vars = 69174 × 49166
      obs: 'orig.ident', 'nCount_RNA', 'nFeature_RNA', 'sample_id', 'percent.mito', 'S.Score', 'G2M.Score', 'Phase', 'CC.Difference', 'nCount_SCT', 'nFeature_SCT', 'age', 'figure_clusters', 'sex', 'type', 'experiment', 'fig_cell_type'
      var: 'name'
      obsm: 'X_pca', 'X_umap', 'X_tsne'
  
  sc.pp.filter_cells(adata, min_genes=500)
  sc.pp.filter_genes(adata, min_cells=10)
  print(adata.n_obs, adata.n_vars)
  
  results_file = 'adata.h5ad'
  adata.write(results_file)
  
  ```
  
  **FULL CODE**
  
  # updated version: h5ad filter, added layers to make the filtered file usable for seurat to read and make a correct seurat object
  
  ```
  #https://github.com/satijalab/seurat/issues/2485
  
  import scanpy as sc
  import numpy as np
  import pandas as pd
  import time
  from ipywidgets import interact, interactive, widgets
  from IPython.display import display, HTML
  
  def filter_and_save_h5ad(h5ad_file, min_genes, min_cells):
      # h5ad file read
      adata = sc.read_h5ad(h5ad_file)
  
      # cells and genes filter parameter from user
      sc.pp.filter_cells(adata, min_genes=min_genes)
      sc.pp.filter_genes(adata, min_cells=min_cells)
  
      # Add 'counts' layer
      adata.layers['counts'] = adata.X.copy()
  
      # add 'winsorization' layer
      adata.layers['winsorized'] = adata.X.copy()
  
      # filtered dataset save
      filtered_file = h5ad_file.replace('.h5ad', '_filtered.h5ad')
      adata.write(filtered_file)
  
      print(f"Filtered dataset saved to {filtered_file}")
  
  # input widgets
  h5ad_file_widget = widgets.Text(description='File_add:')
  min_genes_widget = widgets.IntText(description='Min_Genes:')
  min_cells_widget = widgets.IntText(description='Min_Cells:')
  button = widgets.Button(description='Filter_Save')
  
  # timer widget
  time_widget = widgets.HTML(value="<p>Time passed away: 0 seconds</p>")
  
  def on_button_clicked(click_me):
      h5ad_file = h5ad_file_widget.value.strip()
      min_genes = min_genes_widget.value
      min_cells = min_cells_widget.value
      
      # timer
      start_time = time.time()
      
      filter_and_save_h5ad(h5ad_file, min_genes, min_cells)
      
      # Calculate and update the elapsed time
      elapsed_time = time.time() - start_time
      time_widget.value = f"<p>Time passed away: {elapsed_time} seconds</p>"
  
  button.on_click(on_button_clicked)
  
  # Display the widgets and button
  display(h5ad_file_widget, min_genes_widget, min_cells_widget, button, time_widget)
  
  ```
  
  **Tags:** scrnaseq, single-cell
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2023-05-19. Score: 1 | Accepted answer
  
  You didn't show us how the original .h5ad object was generated, how you read it in, how you wrote it, and if you did anything else in-between besides filtering. [`AnnData.write()`](https://anndata.readthedocs.io/en/latest/generated/anndata.AnnData.write.html) takes a `compression` arg that might explain the difference. As also mentioned in that link, ensuring you are writing a sparse matrix can make a substantial difference to file size (the matrix in `adata.X` is generally by far the largest part of the object)
  
  > **Comments:**
  > **username_0**: I was taking this dataset https://data.humancellatlas.org/explore/projects/85a9263b-0887-48ed-ab1a-ddfa773727b6/get-curl-command  
  > **username_0**: I have updated the question  
  > **username_0**: I have updated my code please have a look and for this "You didn't show us how the original .h5ad object was generated" please have a look https://data.humancellatlas.org/explore/projects/85a9263b-0887-48ed-ab1a-ddfa773727b6  
  
  ---
  
  ### Answer by username_2 on 2023-05-18. Score: 1
  
  Scanpy's `h5ad` is in fact hdf5 format. This format is used for rapid read/write (i/o) large amounts of data and thats its purpose. Its fairly easy to use from scratch.
  
  Its purpose is not to compress data. If you are concerned with the contents of `h5ad` why not just read them directly via the `hdf5` viewer. You can download it here. <https://www.hdfgroup.org/downloads/hdfview/>
  
  The file will then open up as a "Explorer" (Windows) or "Finder" (OSX) GUI file system and you can open the various files and see whats in them.
  
  My suspicion is that the original data is been written into the file. 'cause it can i/o rapidly and access precisely what it needs then file size could be a secondary consideration. Anyway you can peer in and find out by looking at the original file and the hdf5 save you've done.
  
  An example from the website is here
  
  [![enter image description here](https://i.sstatic.net/zhtSf.png)](https://i.sstatic.net/zhtSf.png)
  
  Cool right?
  
  Personally, I think HDF5 is overrated and `feather` outperforms it for speed (i/o) - but thats just for dataframes. Its not much faster than pickling in case you are interested and pickling in addition does compression. parquet is the format that does rapid i/o but specialises in compression, again mostly for dataframes.
  
  HDF5 however does permit very pretty GUI interfaces.
  
  ---
  
  Also if you want to load up and look at various files, that just uses the Python `hdf5` module.
  
  
  ---
  
  