Answers.Body: <pre><code>awk '{print "wget " $9 " -O " $1 "_" $2 ".fastq.gz"}' tmp 
  </code></pre>
  
  <p>will give you </p>
  
  <pre><code>wget https://www.encodeproject.org/files/ENCFF001JPX/@@download/ENCFF001JPX.fastq.gz -O a_k.fastq.gz
  wget https://www.encodeproject.org/files/ENCFF001JPZ/@@download/ENCFF001JPZ.fastq.gz -O b_k.fastq.gz
  wget https://www.encodeproject.org/files/ENCFF001JQB/@@download/ENCFF001JQB.fastq.gz -O c_k.fastq.gz
  </code></pre>
  
  <p>If you add a <code>| bash</code> to the command it will then execute all the <code>wget</code> commands.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-12-13T19:37:45.747Z
Answers.Text: This works great. Can you tell what |bash is doing ??
Answers.UserDisplayName: user3138373
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-12-13T19:43:36.950Z
Answers.Text: @user3138373 the output of the awk command is a set of shell commands, essentially a mini script. By piping it to bash with `| bash` you're telling it to execute each input line as a command. So it will run the `wget` calls and download the sequences.
Answers.UserDisplayName: terdon
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-12-13T19:28:34.387Z
Answers.LastActivityDate: 2017-12-13T19:46:31.503Z
Answers.LastEditDate: 2017-12-13T19:46:31.503Z
Answers.UserDisplayName: Bioathlete
Answers.Body: <p>A variation of <a href="/a/3072/2042">Bioathlete’s answer</a> that doesn't suffer from escaping issues:</p>
  
  
  
  <pre class="lang-sh prettyprint-override"><code>awk 'BEGIN{ FS="\t" } { printf("%s\t%s_%s.fastq.gz\n", $8, $1, $2) }' input |
  while IFS=$'\t' read -r url filename; do
      wget -O "$filename" -- "$url"
  done
  </code></pre>
  
  <p>Or entirely with shell parsing:</p>
  
  <pre class="lang-sh prettyprint-override"><code>while IFS=$'\t' read -r c1 c2 c3 c4 c5 c6 c7 url; do
      wget -O "${c1}_${c2}.fastq.gz" -- "$url"
  done &lt; input
  </code></pre>
  
  <p>Or entirely without shell parsing but with GNU <code>xargs</code>:</p>
  
  <pre class="lang-sh prettyprint-override"><code>awk 'BEGIN{ FS="\t" } { printf("%s_%s.fastq.gz\n%s\n", $1, $2, $8) }' input |
  xargs -r -d '\n' -n 2 -- wget -O
  </code></pre>
  
  <p>… you could even run multiple <code>wget</code> instances in parallel from GNU <code>xargs</code> with the <code>-P</code> option, though you should set the <code>-q</code> option on <code>wget</code> to disable progress bars overwriting each other on the same terminal. Then again it would be better to use a real command-line download manager like Aria or Axel for that task.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-12-13T21:53:00.753Z
Answers.LastActivityDate: 2017-12-13T23:05:17.623Z
Answers.LastEditDate: 2017-12-13T23:05:17.623Z
Answers.UserDisplayName: David Foerster
Body: <p>I have a text file as follows</p>
  
  <pre><code>a   k   testis  adult   male    8 week  rRNA    https://www.encodeproject.org/files/ENCFF001JPX/@@download/ENCFF001JPX.fastq.gz
  b   k   testis  adult   male    8 week  rRNA    https://www.encodeproject.org/files/ENCFF001JPZ/@@download/ENCFF001JPZ.fastq.gz
  c   k   testis  adult   male    8 week  rRNA    https://www.encodeproject.org/files/ENCFF001JQB/@@download/ENCFF001JQB.fastq.gz
  </code></pre>
  
  <p>The file is tab delimited and has 8 columns with 8th column being the url link  I need to download the files one by one using wget and changing the name of the downloaded file with -O option in wget. The argument I want to give to the -O option is combination col1 and col2 separated by _ so that for first line I get the downloaded file as a_k.fastq.gz and so on. For example, the first line of the file above should give:</p>
  
  <pre><code>wget https://www.encodeproject.org/files/ENCFF001JPX/@@download/ENCFF001JPX.fastq.gz -O a_k.fastq.gz
  </code></pre>
  
  <p>Following the post from <a href="https://unix.stackexchange.com/questions/410710/splitting-a-line-into-array-in-bash-with-tab-as-delimiter?noredirect=1#comment735962_410710">here</a> I realized looping through the file is slow and sometimes difficult. How can I download these files using wget with awk or cut as the tool for fixed fields</p>
  
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-12-13T19:20:06.200Z
Comments.Text: Can you post up an example wget command?
Comments.UserDisplayName: Bioathlete
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-12-13T19:23:19.647Z
Comments.Text: wget https://www.encodeproject.org/files/ENCFF001JPX/@@download/ENCFF001JPX.fastq.gz -O a_k.fastq.gz
Comments.UserDisplayName: user3138373
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-12-13T19:23:43.013Z
Comments.Text: how can I put it into code in the comment section??
Comments.UserDisplayName: user3138373
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-12-13T19:25:52.240Z
Comments.Text: edit the original post
Comments.UserDisplayName: Bioathlete
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-12-15T10:35:25.247Z
Comments.Text: @Llopis [we've decided](https://bioinformatics.meta.stackexchange.com/q/88/298) that basic things that bioinformaticians need to do in their daily work are on topic even when the issue could be expressed as a general text parsing question.
Comments.UserDisplayName: terdon
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-12-15T11:21:30.533Z
Comments.Text: Ok, I misjudged and thought it was about downloading data.
Comments.UserDisplayName: llrs
Tags: <awk><data-download>
Title: Downloading URL using AWK for fixed fields
ThreadText: # Downloading URL using AWK for fixed fields
  
  (Asked by: username_0 on 2017-12-13)
  
  I have a text file as follows
  
  ```
  a   k   testis  adult   male    8 week  rRNA    https://www.encodeproject.org/files/ENCFF001JPX/@@download/ENCFF001JPX.fastq.gz
  b   k   testis  adult   male    8 week  rRNA    https://www.encodeproject.org/files/ENCFF001JPZ/@@download/ENCFF001JPZ.fastq.gz
  c   k   testis  adult   male    8 week  rRNA    https://www.encodeproject.org/files/ENCFF001JQB/@@download/ENCFF001JQB.fastq.gz
  
  ```
  
  The file is tab delimited and has 8 columns with 8th column being the url link I need to download the files one by one using wget and changing the name of the downloaded file with -O option in wget. The argument I want to give to the -O option is combination col1 and col2 separated by \_ so that for first line I get the downloaded file as a\_k.fastq.gz and so on. For example, the first line of the file above should give:
  
  ```
  wget https://www.encodeproject.org/files/ENCFF001JPX/@@download/ENCFF001JPX.fastq.gz -O a_k.fastq.gz
  
  ```
  
  Following the post from [here](https://unix.stackexchange.com/questions/410710/splitting-a-line-into-array-in-bash-with-tab-as-delimiter?noredirect=1#comment735962_410710) I realized looping through the file is slow and sometimes difficult. How can I download these files using wget with awk or cut as the tool for fixed fields
  
  **Tags:** awk, data-download
  
  > **Comments:**
  > **username_1**: Can you post up an example wget command?  
  > **username_0**: wget https://www.encodeproject.org/files/ENCFF001JPX/@@download/ENCFF001JPX.fastq.gz -O a_k.fastq.gz  
  > **username_0**: how can I put it into code in the comment section??  
  > **username_1**: edit the original post  
  > **username_2**: @Llopis [we've decided](https://bioinformatics.meta.stackexchange.com/q/88/298) that basic things that bioinformaticians need to do in their daily work are on topic even when the issue could be expressed as a general text parsing question.  
  > **username_3**: Ok, I misjudged and thought it was about downloading data.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2017-12-13. Score: 5
  
  ```
  awk '{print "wget " $9 " -O " $1 "_" $2 ".fastq.gz"}' tmp 
  
  ```
  
  will give you
  
  ```
  wget https://www.encodeproject.org/files/ENCFF001JPX/@@download/ENCFF001JPX.fastq.gz -O a_k.fastq.gz
  wget https://www.encodeproject.org/files/ENCFF001JPZ/@@download/ENCFF001JPZ.fastq.gz -O b_k.fastq.gz
  wget https://www.encodeproject.org/files/ENCFF001JQB/@@download/ENCFF001JQB.fastq.gz -O c_k.fastq.gz
  
  ```
  
  If you add a `| bash` to the command it will then execute all the `wget` commands.
  
  > **Comments:**
  > **username_0**: This works great. Can you tell what |bash is doing ??  
  > **username_2**: @username_0 the output of the awk command is a set of shell commands, essentially a mini script. By piping it to bash with `| bash` you're telling it to execute each input line as a command. So it will run the `wget` calls and download the sequences.  
  
  ---
  
  ### Answer by username_4 on 2017-12-13. Score: 4
  
  A variation of [username_1’s answer](/a/3072/2042) that doesn't suffer from escaping issues:
  
  ```sh
  awk 'BEGIN{ FS="\t" } { printf("%s\t%s_%s.fastq.gz\n", $8, $1, $2) }' input |
  while IFS=$'\t' read -r url filename; do
      wget -O "$filename" -- "$url"
  done
  
  ```
  
  Or entirely with shell parsing:
  
  ```sh
  while IFS=$'\t' read -r c1 c2 c3 c4 c5 c6 c7 url; do
      wget -O "${c1}_${c2}.fastq.gz" -- "$url"
  done < input
  
  ```
  
  Or entirely without shell parsing but with GNU `xargs`:
  
  ```sh
  awk 'BEGIN{ FS="\t" } { printf("%s_%s.fastq.gz\n%s\n", $1, $2, $8) }' input |
  xargs -r -d '\n' -n 2 -- wget -O
  
  ```
  
  … you could even run multiple `wget` instances in parallel from GNU `xargs` with the `-P` option, though you should set the `-q` option on `wget` to disable progress bars overwriting each other on the same terminal. Then again it would be better to use a real command-line download manager like Aria or Axel for that task.
  
  
  ---
  
  