Answers.Body: <p>I would suggest subsampling an existing dataset such as the <a href="https://www.nist.gov/programs-projects/genome-bottle" rel="nofollow noreferrer">GIAB</a> resource that you've already indicated.</p>
  <p>Use a utility such as <a href="https://github.com/lh3/seqtk" rel="nofollow noreferrer">seqtk</a> to subsample the appropriate number of reads from the dataset, and use that.</p>
  <p>Example with <code>seqtk</code> to get 1 million reads each from paired fastqs while preserving pairing:</p>
  <pre><code>  # note -s100 random seed which should be used for both to keep pairs
    seqtk sample -s100 giab_read1.fq 1000000 &gt; giab_sub_r1.fq
    seqtk sample -s100 giab_read2.fq 1000000 &gt; giab_sub_r2.fq
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-30T02:46:42.517Z
Answers.Text: Ok, that's actually a good suggestion. However, how to subsample gold standard fastq, .bam, and .vcf? So, that I know my pipeline steps/changes are okay in all three steps?
Answers.UserDisplayName: Shafayet Rahat
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-30T09:31:23.533Z
Answers.Text: @ShafayetRahat just subsample the fastq, and then run the small fastq files through your pipeline to get bam abd vcf. I use a similar approach, and it only takes about 5 minutes to run through the pipeline with these small files.
Answers.UserDisplayName: terdon
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-30T19:42:33.407Z
Answers.Text: @ShafayetRahat terdon is right, unless you're doing something odd mapping should be unaffected by depth. And you will not be able to subsample a VCF to only represent certain reads, I am not sure how you would do that. (though in principle you can do it with a BAM file, it would just be a lot of work compared to re-mapping).
Answers.UserDisplayName: Maximilian Press
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-29T21:08:03.470Z
Answers.LastActivityDate: 2023-10-29T21:08:03.470Z
Answers.UserDisplayName: Maximilian Press
Body: <p>I am trying to check my variant generation pipeline for humans. The pipeline is for NGS. I am searching for any dataset(.fastq+.bam+.vcf) for any specific sample. Now, GIAB has a nice dataset for that. However, the problem is that it's not a tiny dataset. And it's not feasible to test my pipeline with this data every time. I tried to generate a synthetic dataset with some read simulators(NEAT, MASON2). However, their aligned read files(.bam) files seem not very good. You can check my Issue for NEAT in <a href="https://github.com/ncsa/NEAT/issues/88" rel="nofollow noreferrer">link</a>.
  So, my question is, <strong>Is there any publicly available tiny dataset(.fastq,.bam, .vcf) with which I can validate my pipeline fast? Or, is there any simulator with which I can simulate .bam, .fastq files from a given input .vcf? I am using fastqc,bwa,samtools,GATK4 mainly and GRCH38 reference in my pipeline.</strong></p>
  
Tags: <ngs><public-dataset><simulated-data><validation>
Title: Generating simulated gold standard NGS dataset/ Any publicly available smal gold standard NGS dataset
ThreadText: # Generating simulated gold standard NGS dataset/ Any publicly available smal gold standard NGS dataset
  
  (Asked by: username_0 on 2023-10-29)
  
  I am trying to check my variant generation pipeline for humans. The pipeline is for NGS. I am searching for any dataset(.fastq+.bam+.vcf) for any specific sample. Now, GIAB has a nice dataset for that. However, the problem is that it's not a tiny dataset. And it's not feasible to test my pipeline with this data every time. I tried to generate a synthetic dataset with some read simulators(NEAT, MASON2). However, their aligned read files(.bam) files seem not very good. You can check my Issue for NEAT in [link](https://github.com/ncsa/NEAT/issues/88).
  So, my question is, **Is there any publicly available tiny dataset(.fastq,.bam, .vcf) with which I can validate my pipeline fast? Or, is there any simulator with which I can simulate .bam, .fastq files from a given input .vcf? I am using fastqc,bwa,samtools,GATK4 mainly and GRCH38 reference in my pipeline.**
  
  **Tags:** ngs, public-dataset, simulated-data, validation
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-10-29. Score: 2
  
  I would suggest subsampling an existing dataset such as the [GIAB](https://www.nist.gov/programs-projects/genome-bottle) resource that you've already indicated.
  
  Use a utility such as [seqtk](https://github.com/lh3/seqtk) to subsample the appropriate number of reads from the dataset, and use that.
  
  Example with `seqtk` to get 1 million reads each from paired fastqs while preserving pairing:
  
  ```
    # note -s100 random seed which should be used for both to keep pairs
    seqtk sample -s100 giab_read1.fq 1000000 > giab_sub_r1.fq
    seqtk sample -s100 giab_read2.fq 1000000 > giab_sub_r2.fq
  
  ```
  
  > **Comments:**
  > **username_0**: Ok, that's actually a good suggestion. However, how to subsample gold standard fastq, .bam, and .vcf? So, that I know my pipeline steps/changes are okay in all three steps?  
  > **username_2**: @username_0 just subsample the fastq, and then run the small fastq files through your pipeline to get bam abd vcf. I use a similar approach, and it only takes about 5 minutes to run through the pipeline with these small files.  
  > **username_1**: @username_0 username_2 is right, unless you're doing something odd mapping should be unaffected by depth. And you will not be able to subsample a VCF to only represent certain reads, I am not sure how you would do that. (though in principle you can do it with a BAM file, it would just be a lot of work compared to re-mapping).  
  
  ---
  
  