Answers.Body: <p>I am not NGS expert, I known ML. Essentially you are performing data augmentation which is essential for ML. What I question is whether a replicate of e6 from a population of e7 would really be sufficient for ML because you need sample sizes (replicates in this case) of >100000. </p>
  
  <p>The answer is really simple you bootstrap the data (sample with replacement), to obtain your replicates. Numpy has a random module that will do this,</p>
  
  <pre><code>p.random.seed(345)
  pop = np.random.randint(0,10000000 , size=1000000)
  </code></pre>
  
  <p>Use <code>pop</code> to select from fastaq. The other way in is to define each fastq sequence within pop then simply,</p>
  
  <pre><code>sample = np.random.choice(pop, size=1000000)
  </code></pre>
  
  <p>Or a combination of the two. If you can do ML you can do Python coding.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-15T21:45:46.323Z
Answers.Text: Dear Michael, thank you for the answer, but what I am asking is not how to do random subsampling but the idea whether it makes sense to do it for metagenomic data where it may remove key information of rare taxons. I don't think that subsampling should be done with replacement anyway since the subsamples would not be considered independent for training and testing.
Answers.UserDisplayName: Remy
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-15T23:56:07.233Z
Answers.Text: Ok, in that case I don't understand the rationale. Sampling is used heavily in machine learning, it is also used in statistical analysis such as Monte Carlo and  null distributions. Generally it is used to ensure the observation is an outlier (except for ML)
Answers.UserDisplayName: M__
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-13T04:49:07.583Z
Answers.LastActivityDate: 2020-03-13T04:49:07.583Z
Answers.UserDisplayName: M__
Body: <p>Let's say we have a metagenomic sample that is paired-end FASTQ files including 10,000,000 DNA reads collected using shotgun sequencing. 
  How would one make a random subsample of the mentioned metagenomic sample with for example 1,000,000 reads? I know about the <code>seqtk</code> function that could be used in such cases, but my question is more about whether it is meaningful to represent a metagenomic sample with a small subsample generated using this way or it has to satisfy certain criteria, e.g. in terms of the number of reads and diversity to be a valid representative of the metagenomic sample? 
  Knowing that some species might be very rare in the sample, would it ever be proper to do subsampling on metagenomic samples? </p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-03-09T12:00:49.833Z
Comments.Text: To make more samples to be used for machine learning
Comments.UserDisplayName: Remy
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-03-13T04:35:52.093Z
Comments.Text: Ah ... data augmentation, yep thats a common approach. I personally can't quite see what teh problem is, if you can perform ML surely you can use a randomisation module, e.g. Python's random or sample?
Comments.UserDisplayName: M__
Tags: <phylogenetics><phylogeny><fastq><taxonomy><microbiology>
Title: What is a proper way for random subsampling of metagenomic data?
ThreadText: # What is a proper way for random subsampling of metagenomic data?
  
  (Asked by: username_0 on 2020-03-09)
  
  Let's say we have a metagenomic sample that is paired-end FASTQ files including 10,000,000 DNA reads collected using shotgun sequencing.
  How would one make a random subsample of the mentioned metagenomic sample with for example 1,000,000 reads? I know about the `seqtk` function that could be used in such cases, but my question is more about whether it is meaningful to represent a metagenomic sample with a small subsample generated using this way or it has to satisfy certain criteria, e.g. in terms of the number of reads and diversity to be a valid representative of the metagenomic sample?
  Knowing that some species might be very rare in the sample, would it ever be proper to do subsampling on metagenomic samples?
  
  **Tags:** phylogenetics, phylogeny, fastq, taxonomy, microbiology
  
  > **Comments:**
  > **username_0**: To make more samples to be used for machine learning  
  > **username_1**: Ah ... data augmentation, yep thats a common approach. I personally can't quite see what teh problem is, if you can perform ML surely you can use a randomisation module, e.g. Python's random or sample?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-03-13. Score: 1
  
  I am not NGS expert, I known ML. Essentially you are performing data augmentation which is essential for ML. What I question is whether a replicate of e6 from a population of e7 would really be sufficient for ML because you need sample sizes (replicates in this case) of >100000.
  
  The answer is really simple you bootstrap the data (sample with replacement), to obtain your replicates. Numpy has a random module that will do this,
  
  ```
  p.random.seed(345)
  pop = np.random.randint(0,10000000 , size=1000000)
  
  ```
  
  Use `pop` to select from fastaq. The other way in is to define each fastq sequence within pop then simply,
  
  ```
  sample = np.random.choice(pop, size=1000000)
  
  ```
  
  Or a combination of the two. If you can do ML you can do Python coding.
  
  > **Comments:**
  > **username_0**: Dear Michael, thank you for the answer, but what I am asking is not how to do random subsampling but the idea whether it makes sense to do it for metagenomic data where it may remove key information of rare taxons. I don't think that subsampling should be done with replacement anyway since the subsamples would not be considered independent for training and testing.  
  > **username_1**: Ok, in that case I don't understand the rationale. Sampling is used heavily in machine learning, it is also used in statistical analysis such as Monte Carlo and  null distributions. Generally it is used to ensure the observation is an outlier (except for ML)  
  
  ---
  
  