Answers.Body: <p>Not sure why you would want to do this, but if the aim is to repeat <strong>all</strong> records in the fastq/fasta files <code>n</code> times, the most efficient way I can think of is just to concatenate the orginal file <code>n</code> times, similar to <a href="https://superuser.com/questions/338616/linux-command-to-concatenate-a-file-to-itself-n-times">this thread</a>.</p>
  <p>i.e adapting the accepted answer would look something like this (I have not tested this):</p>
  <pre><code>for i in {1..n};do cat file.fq &gt;&gt; file_ntimes.fq ; done
  </code></pre>
  <p><strong>Update based on comments:</strong></p>
  <p>To replicate each fastq entry back to back, maybe this hacky unix command can be adjusted (say for n=4):</p>
  <pre><code>n=10
  paste - - - - &lt;file.fq | awk '{for(i=0;i&lt;$n;i++)print}' | tr '\t' '\n' &gt; file_10times.fq
  </code></pre>
  <p>The idea is to <code>paste</code> blocks of 4 lines (i.e. 1 fastq entry) into one line and replicate this line n times. <code>tr</code> is reversing the paste action (since the separation by paste is a tab character). This won't work if fastq file for some reason has tabs in it which I don't expect but didn't test either.
  Note, for fasta file this would be <code>paste - -</code> rather than <code>paste - - - -</code></p>
  <p>I haven't tested this, but if it works for you I could polish it up if needed.</p>
  <p><strong>Update2</strong></p>
  <p>Based on Xioafeis comment this is solution seems to work as well:</p>
  <pre><code>bioawk -cfastx '{for(i=1; i&lt;=10; i++) {print &quot;@&quot;$name&quot;\n&quot;$seq&quot;\n+\n&quot;$qual}}' file.fq &gt; file_10times.fq
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-17T13:28:38.767Z
Answers.Text: Thank you so much for your reply!
  But I don't think I made my question clear. I want to repeat rd_1 for 100 times, rd_2 for 100 times, ... in this order, not (rd_1, rd_2, ...) x 100 times.
Answers.UserDisplayName: Xiaofei Wang
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-17T13:36:54.300Z
Answers.Text: My goal is to pairwise each of read record in file_1.fq with each of record in file_2.fq into 2 new fastq files and get overlap between them. Actually, for the one that you suggested that is what I want to for my file_new_2.fq, which I have already figured out. But not the one that I am asking. I will edit my question with a figure. Really appreciate for your reply!
Answers.UserDisplayName: Xiaofei Wang
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-17T13:43:15.863Z
Answers.Text: I did wrote a python code using biopython, but it is very slow I guess it slurped whole file into dictionary, so very very slow and seems running forever
Answers.UserDisplayName: Xiaofei Wang
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-18T12:05:35.623Z
Answers.Text: Thank you so much! I figured it out by using bioawk.
Answers.UserDisplayName: Xiaofei Wang
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-18T12:08:58.673Z
Answers.Text: `bioawk -cfastx '{for(i=1; i<=10; i++) {print "@"$name"\n"$seq"\n+\n"$qual}}' test_repeat.fq > test_repeat_10times.fq`
Answers.UserDisplayName: Xiaofei Wang
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-18T12:59:51.157Z
Answers.Text: I see, did my suggested solution work as well? Feel free to upvote or accept the answer in that case! Also Happy to add you solution as 3rd option if you want to have it as offical answer?
Answers.UserDisplayName: Sebastian Müller
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-17T12:30:12.000Z
Answers.LastActivityDate: 2021-01-20T08:07:29.723Z
Answers.LastEditDate: 2021-01-20T08:07:29.723Z
Answers.UserDisplayName: Sebastian Müller
Body: <p><a href="https://i.sstatic.net/Qdrm0.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Qdrm0.jpg" alt="enter image description here" /></a>How to iterate/repeat a record n times in a fastq file using bioawk? I wrote a python code using biopython, but it is very very slow. So, I am wondering if I can get some help by using bioawk. Thank you so much!</p>
  <p>for example, in the file.fq file, I'd like to repeat read_id1 for 100times, read_id2 for 100times, ..., and write them to a new file.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-17T12:43:05.820Z
Comments.Text: Hi and welcom to BS, could you please a little bit more what you problem you want to solve? Do you want only specific record or all records of the file? Maybe you can show your python code for clarity.
Comments.UserDisplayName: Mr_Z
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-17T18:29:25.290Z
Comments.Text: You might want to show your biopython implementation. While biopython can be slow, your solution may be sub-optimal
Comments.UserDisplayName: Chris_Rands
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-22T19:03:27.927Z
Comments.Text: There are quite a few people complaining about the speed of Biopython... I think [pysam](https://pysam.readthedocs.io/en/latest/api.html#fastq-files) is much faster.
Comments.UserDisplayName: Kamil S Jaron
Tags: <fastq><shell><perl><bioawk><repeat>
Title: how to repeat records in fastq n times efficiently?
ThreadText: # how to repeat records in fastq n times efficiently?
  
  (Asked by: username_0 on 2021-01-17)
  
  [![enter image description here](https://i.sstatic.net/Qdrm0.jpg)](https://i.sstatic.net/Qdrm0.jpg)How to iterate/repeat a record n times in a fastq file using bioawk? I wrote a python code using biopython, but it is very very slow. So, I am wondering if I can get some help by using bioawk. Thank you so much!
  
  for example, in the file.fq file, I'd like to repeat read\_id1 for 100times, read\_id2 for 100times, ..., and write them to a new file.
  
  **Tags:** fastq, shell, perl, bioawk, repeat
  
  > **Comments:**
  > **username_1**: Hi and welcom to BS, could you please a little bit more what you problem you want to solve? Do you want only specific record or all records of the file? Maybe you can show your python code for clarity.  
  > **username_2**: You might want to show your biopython implementation. While biopython can be slow, your solution may be sub-optimal  
  > **username_3**: There are quite a few people complaining about the speed of Biopython... I think [pysam](https://pysam.readthedocs.io/en/latest/api.html#fastq-files) is much faster.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_4 on 2021-01-17. Score: 0 | Accepted answer
  
  Not sure why you would want to do this, but if the aim is to repeat **all** records in the fastq/fasta files `n` times, the most efficient way I can think of is just to concatenate the orginal file `n` times, similar to [this thread](https://superuser.com/questions/338616/linux-command-to-concatenate-a-file-to-itself-n-times).
  
  i.e adapting the accepted answer would look something like this (I have not tested this):
  
  ```
  for i in {1..n};do cat file.fq >> file_ntimes.fq ; done
  
  ```
  
  **Update based on comments:**
  
  To replicate each fastq entry back to back, maybe this hacky unix command can be adjusted (say for n=4):
  
  ```
  n=10
  paste - - - - <file.fq | awk '{for(i=0;i<$n;i++)print}' | tr '\t' '\n' > file_10times.fq
  
  ```
  
  The idea is to `paste` blocks of 4 lines (i.e. 1 fastq entry) into one line and replicate this line n times. `tr` is reversing the paste action (since the separation by paste is a tab character). This won't work if fastq file for some reason has tabs in it which I don't expect but didn't test either.
  Note, for fasta file this would be `paste - -` rather than `paste - - - -`
  
  I haven't tested this, but if it works for you I could polish it up if needed.
  
  **Update2**
  
  Based on Xioafeis comment this is solution seems to work as well:
  
  ```
  bioawk -cfastx '{for(i=1; i<=10; i++) {print "@"$name"\n"$seq"\n+\n"$qual}}' file.fq > file_10times.fq
  
  ```
  
  > **Comments:**
  > **username_0**: Thank you so much for your reply! But I don't think I made my question clear. I want to repeat rd_1 for 100 times, rd_2 for 100 times, ... in this order, not (rd_1, rd_2, ...) x 100 times.  
  > **username_0**: My goal is to pairwise each of read record in file_1.fq with each of record in file_2.fq into 2 new fastq files and get overlap between them. Actually, for the one that you suggested that is what I want to for my file_new_2.fq, which I have already figured out. But not the one that I am asking. I will edit my question with a figure. Really appreciate for your reply!  
  > **username_0**: I did wrote a python code using biopython, but it is very slow I guess it slurped whole file into dictionary, so very very slow and seems running forever  
  > **username_0**: Thank you so much! I figured it out by using bioawk.  
  > **username_0**: `bioawk -cfastx '{for(i=1; i<=10; i++) {print "@"$name"\n"$seq"\n+\n"$qual}}' test_repeat.fq > test_repeat_10times.fq`  
  > **username_4**: I see, did my suggested solution work as well? Feel free to upvote or accept the answer in that case! Also Happy to add you solution as 3rd option if you want to have it as offical answer?  
  
  ---
  
  