Answers.Body: <p>Something like this might do it:</p>
  
  <pre><code>#!/bin/bash
  
  # make an array to hold your list of samples/base filenames
  samples=( S1 S2 S3 S4 S5 )
  
  # make a loop to go through the sample IDs
  for sample in "<span class="math-container">${samples[@]}"
  do
    :
    # do something with each sample/file, e.g. make a directory for it
    mkdir "$</span>{sample}"_home
  
    # do something else with the sample, e.g. unzip the fastq.gz file
    gunzip "<span class="math-container">${sample}".fq.gz &gt; "$</span>{sample}"_home/"${sample}".fq
  done
  </code></pre>
  
  <p>One disadvantage with this is that it will process the files one after the other.  If you have a cluster, you might prefer to run them all at the same time to make the whole thing quicker.  If you have a SLURM cluster, you can use job array to do something similar.</p>
  
  <p>If you have your list of samples in a text file, you could try replacing the samples array command in the script above with something like this:</p>
  
  <pre><code>samples=( `cat samples_list.txt` )
  </code></pre>
  
  <p>This will only work if the samples are listed on a single line, space-separated. You may need to process your samples file further to get them in this format, or use the bash readarray command.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-12-11T13:14:57.370Z
Answers.Text: I have a list of sample ids store in a txt so is it possible that it make a list using that file?
Answers.UserDisplayName: Safina A.R
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-12-11T13:26:29.720Z
Answers.Text: @SafinaA.R please remember to actually show your data. Edit your question and show us this text file. That way, we will know the format and can use it to give you example answers.
Answers.UserDisplayName: terdon
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-12-11T16:55:07.897Z
Answers.Text: This solution requires saving the sample names directly into the bash script. This sounds a little difficult since OP said she has a file with 4000 sample names.
Answers.UserDisplayName: conchoecia
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-12-12T04:16:27.970Z
Answers.Text: hi @terdon
  The data is same as shown by fin swimmer in the above example. It is a text file tab separated having two col. Sample Ids and Library Ids
Answers.UserDisplayName: Safina A.R
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-01-04T16:30:54.430Z
Answers.Text: I haven't tried this, but I added a comment to my script above about how to get the sample IDs out of a text file.
Answers.UserDisplayName: Jay Moore
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-12-11T12:29:47.003Z
Answers.LastActivityDate: 2019-01-04T16:39:30.057Z
Answers.LastEditDate: 2019-01-04T16:39:30.057Z
Answers.UserDisplayName: Jay Moore
Answers.Body: <p>if you have a repeating workflow, I strongly recommend to have a look at workflow management systems like <a href="https://snakemake.readthedocs.io/en/stable/tutorial/basics.html" rel="noreferrer">snakemake</a>. I've also wrote a little tutorial on <a href="https://www.biostars.org/p/335903/" rel="noreferrer">biostars</a> about this topic, which might be useful for you.</p>
  
  <p>fin swimmer</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-12-11T11:04:34.193Z
Answers.Text: That's a bit complicated can there be simple for loop trick?
Answers.UserDisplayName: Safina A.R
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-12-11T10:51:24.073Z
Answers.LastActivityDate: 2018-12-11T10:51:24.073Z
Answers.UserDisplayName: finswimmer
Answers.Body: <p>You can create a script to pass arguments to as follows:</p>
  
  <pre><code>#! /bin/bash
  
  bwa mem -M -R "@RG\tID:<span class="math-container">${1}\tLB:$</span>{2}\tPL:ILLUMINA\tPM:HiSeq2000\tSM:<span class="math-container">${1}" ../ref/M._tuberculosis_H37Rv_2015-11-13.fasta $</span>{1}_* &gt; ${1}.sam
  </code></pre>
  
  <p>Now if you run <code>bash script.sh sample lib</code> then <code>sample</code> and <code>lib</code> will be  the script as arguments <code><span class="math-container">${1}</code> and <code>$</span>{2}</code> respectively. Then all you need is a loop to pass arguments to the script:</p>
  
  <pre><code>for sample in sample1 sample2 sample3
     do
     for lib in lib1 lib2 lib3
       do
       bash script.sh <span class="math-container">$sample $</span>lib
       done
     done
  </code></pre>
  
  <p>You can use <code>nohup</code> to run each sample in the background and proceed with next one. This will in essence, submit each command as a separate job to run on a different core. These will print to separate log files sample.out instead of to the terminal. It will also continue to run, even if you disconnect from SSH or close the terminal.</p>
  
  <pre><code>for sample in sample1 sample2 sample3
     do
     for lib in lib1 lib2 lib3
       do
       nohup bash script.sh <span class="math-container">$sample $</span>lib &gt; ${sample}.out &amp;
       done
     done
  </code></pre>
  
  <p>You should only do this for single-threaded processes. For multi-threaded processes or memory-intensive jobs, this will demand a lot from the machine. Even if this isn’t a shared server, it will perform poorly if you run out of memory and have to “swap” memory by writing to disk. So it’s best to submit them one-by-one (above) or use a cluster with a job scheduler. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-01-05T01:57:30.633Z
Answers.LastActivityDate: 2019-01-05T01:57:30.633Z
Answers.UserDisplayName: Tom Kelly ケリー・トム
Answers.Body: <p>Another solution for repeating one command with different parameters is to use <code>parallel</code>.</p>
  
  <p>Create a tab-delimited file <code>samples.txt</code>, which contains the sample names and the information for your read groups, e.g.:</p>
  
  <pre><code>Sample1 Lib1
  Sample2 Lib2
  Sample3 Lib3
  </code></pre>
  
  <p>Then you can use <code>parallel</code> like this:</p>
  
  <pre><code>parallel --colsep "\t" 'bwa mem -M -R "@RG\tID:{1}\tLB:{2}\tPL:ILLUMINA\tPM:HiSeq2000\tSM:{1}" ../ref/M._tuberculosis_H37Rv_2015-11-13.fasta {1}_* &gt; {1}.sam' :::: samples.txt
  </code></pre>
  
  <p>For each line in <code>samples.txt</code> a new process is started, replacing <code>{1}</code> and <code>{2}</code> with the values in column 1 and 2 of this line.</p>
  
  <p>fin swimmer</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-12-11T13:15:43.897Z
Answers.Text: wow that's look much easier i will try and update you.
Answers.UserDisplayName: Safina A.R
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-12-12T07:57:33.827Z
Answers.Text: swimmer thank you for this simplest code. I want to know whether this parallel produce log? because i cant see anything on the screen as the bwa command produce screen messages while running. Secondly, if i don't want to run these commands parallel than what should i used instead of parallel as while running it parallel my computer got stuck.
Answers.UserDisplayName: Safina A.R
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-12-12T08:57:51.387Z
Answers.Text: By default `parallel` prints to the screen after a job has finished. Doing this makes sure that you can e.g. concatenate outputs of parallel jobs without mixing them. You can use the option `--ungroup` to get the output immediately. But use this with care! Without any parameter the number of parallel jobs is the same as the number of CPU cores. You can set it to an explicit number using `-j`, e.g. `-j 1`
Answers.UserDisplayName: finswimmer
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-12-12T12:35:26.113Z
Answers.Text: Ok. Thank you. It is running for this command i will modified it accordingly for other steps and if i got stuck i will update here.
Answers.UserDisplayName: Safina A.R
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-12-11T13:08:13.020Z
Answers.LastActivityDate: 2018-12-11T13:08:13.020Z
Answers.UserDisplayName: finswimmer
Body: <p>Hi im doing Variant calling on fastq files for which i have 4000 fq files and the variant calling are done in different 9 steps. Each step generate different files that are output to next steps.</p>
  <p>So can anybody help me with this to do it in a loop it pick up names from previous steps for all files. Step 1 command is here:</p>
  <pre><code>bwa mem -M -R '@RG\tID:ERR040140\tLB:2517707\tPL:ILLUMINA\tPM:HiSeq2000\tSM:ERR040140' ../ref/M._tuberculosis_H37Rv_2015-11-13.fasta ERR040140_* &gt; ERR040140.sam
  </code></pre>
  <p>So i made an excel file to break the command in columns so that i can change the file names keeping remaining command same just like this:</p>
  <p><a href="https://i.sstatic.net/uj6vl.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/uj6vl.png" alt="command_column_wise" /></a></p>
  <p>So changes are occurring 5th, 7th and 9th column. which are:</p>
  <pre><code>tID:ERR040140
  tLB:2517707
  tSM:ERR040140
  ERR040140_*
  ERR040140.sam
  </code></pre>
  <p>Can anyone help me with loop or something?</p>
  <p>Thank you!</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-12-11T11:13:29.083Z
Comments.Text: Please [edit] your question and replace the image of the excel file with a copy/paste of the same file exported to tsv (save as 'tab separated text file'). That way we can actually copy it and use it to test our examples.
Comments.UserDisplayName: terdon
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-12-11T18:38:08.243Z
Comments.Text: Please remember to confirm an answer once you've received one.
Comments.UserDisplayName: Stephopolis
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-12-12T04:23:15.190Z
Comments.Text: I have already given the excel file command above. And im unable to paste it here the page is not letting me paste it that is why i quoted 1 example and than added a picture with all other repeating commands with different file names.
Comments.UserDisplayName: Safina A.R
Tags: <variant-calling><linux>
Title: How to run same command on multiple files?
ThreadText: # How to run same command on multiple files?
  
  (Asked by: username_0 on 2018-12-11)
  
  Hi im doing Variant calling on fastq files for which i have 4000 fq files and the variant calling are done in different 9 steps. Each step generate different files that are output to next steps.
  
  So can anybody help me with this to do it in a loop it pick up names from previous steps for all files. Step 1 command is here:
  
  ```
  bwa mem -M -R '@RG\tID:ERR040140\tLB:2517707\tPL:ILLUMINA\tPM:HiSeq2000\tSM:ERR040140' ../ref/M._tuberculosis_H37Rv_2015-11-13.fasta ERR040140_* > ERR040140.sam
  
  ```
  
  So i made an excel file to break the command in columns so that i can change the file names keeping remaining command same just like this:
  
  [![command_column_wise](https://i.sstatic.net/uj6vl.png)](https://i.sstatic.net/uj6vl.png)
  
  So changes are occurring 5th, 7th and 9th column. which are:
  
  ```
  tID:ERR040140
  tLB:2517707
  tSM:ERR040140
  ERR040140_*
  ERR040140.sam
  
  ```
  
  Can anyone help me with loop or something?
  
  Thank you!
  
  **Tags:** variant-calling, linux
  
  > **Comments:**
  > **username_1**: Please [edit] your question and replace the image of the excel file with a copy/paste of the same file exported to tsv (save as 'tab separated text file'). That way we can actually copy it and use it to test our examples.  
  > **username_2**: Please remember to confirm an answer once you've received one.  
  > **username_0**: I have already given the excel file command above. And im unable to paste it here the page is not letting me paste it that is why i quoted 1 example and than added a picture with all other repeating commands with different file names.  
  
  ---
  
  ## 4 answers
  
  ### Answer by username_5 on 2018-12-11. Score: 10 | Accepted answer
  
  Another solution for repeating one command with different parameters is to use `parallel`.
  
  Create a tab-delimited file `samples.txt`, which contains the sample names and the information for your read groups, e.g.:
  
  ```
  Sample1 Lib1
  Sample2 Lib2
  Sample3 Lib3
  
  ```
  
  Then you can use `parallel` like this:
  
  ```
  parallel --colsep "\t" 'bwa mem -M -R "@RG\tID:{1}\tLB:{2}\tPL:ILLUMINA\tPM:HiSeq2000\tSM:{1}" ../ref/M._tuberculosis_H37Rv_2015-11-13.fasta {1}_* > {1}.sam' :::: samples.txt
  
  ```
  
  For each line in `samples.txt` a new process is started, replacing `{1}` and `{2}` with the values in column 1 and 2 of this line.
  
  fin swimmer
  
  > **Comments:**
  > **username_0**: wow that's look much easier i will try and update you.  
  > **username_0**: swimmer thank you for this simplest code. I want to know whether this parallel produce log? because i cant see anything on the screen as the bwa command produce screen messages while running. Secondly, if i don't want to run these commands parallel than what should i used instead of parallel as while running it parallel my computer got stuck.  
  > **username_5**: By default `parallel` prints to the screen after a job has finished. Doing this makes sure that you can e.g. concatenate outputs of parallel jobs without mixing them. You can use the option `--ungroup` to get the output immediately. But use this with care! Without any parameter the number of parallel jobs is the same as the number of CPU cores. You can set it to an explicit number using `-j`, e.g. `-j 1`  
  > **username_0**: Ok. Thank you. It is running for this command i will modified it accordingly for other steps and if i got stuck i will update here.  
  
  ---
  
  ### Answer by username_3 on 2018-12-11. Score: 8
  
  Something like this might do it:
  
  ```
  #!/bin/bash
  
  # make an array to hold your list of samples/base filenames
  samples=( S1 S2 S3 S4 S5 )
  
  # make a loop to go through the sample IDs
  for sample in "${samples[@]}"
  do
    :
    # do something with each sample/file, e.g. make a directory for it
    mkdir "${sample}"_home
  
    # do something else with the sample, e.g. unzip the fastq.gz file
    gunzip "${sample}".fq.gz > "${sample}"_home/"${sample}".fq
  done
  
  ```
  
  One disadvantage with this is that it will process the files one after the other. If you have a cluster, you might prefer to run them all at the same time to make the whole thing quicker. If you have a SLURM cluster, you can use job array to do something similar.
  
  If you have your list of samples in a text file, you could try replacing the samples array command in the script above with something like this:
  
  ```
  samples=( `cat samples_list.txt` )
  
  ```
  
  This will only work if the samples are listed on a single line, space-separated. You may need to process your samples file further to get them in this format, or use the bash readarray command.
  
  > **Comments:**
  > **username_0**: I have a list of sample ids store in a txt so is it possible that it make a list using that file?  
  > **username_1**: @username_0 please remember to actually show your data. Edit your question and show us this text file. That way, we will know the format and can use it to give you example answers.  
  > **username_4**: This solution requires saving the sample names directly into the bash script. This sounds a little difficult since OP said she has a file with 4000 sample names.  
  > **username_0**: hi @username_1 The data is same as shown by fin swimmer in the above example. It is a text file tab separated having two col. Sample Ids and Library Ids  
  > **username_3**: I haven't tried this, but I added a comment to my script above about how to get the sample IDs out of a text file.  
  
  ---
  
  ### Answer by username_5 on 2018-12-11. Score: 5
  
  if you have a repeating workflow, I strongly recommend to have a look at workflow management systems like [snakemake](https://snakemake.readthedocs.io/en/stable/tutorial/basics.html). I've also wrote a little tutorial on [biostars](https://www.biostars.org/p/335903/) about this topic, which might be useful for you.
  
  fin swimmer
  
  > **Comments:**
  > **username_0**: That's a bit complicated can there be simple for loop trick?  
  
  ---
  
  ### Answer by username_6 on 2019-01-05. Score: 0
  
  You can create a script to pass arguments to as follows:
  
  ```
  #! /bin/bash
  
  bwa mem -M -R "@RG\tID:${1}\tLB:${2}\tPL:ILLUMINA\tPM:HiSeq2000\tSM:${1}" ../ref/M._tuberculosis_H37Rv_2015-11-13.fasta ${1}_* > ${1}.sam
  
  ```
  
  Now if you run `bash script.sh sample lib` then `sample` and `lib` will be the script as arguments `${1}` and `${2}` respectively. Then all you need is a loop to pass arguments to the script:
  
  ```
  for sample in sample1 sample2 sample3
     do
     for lib in lib1 lib2 lib3
       do
       bash script.sh $sample $lib
       done
     done
  
  ```
  
  You can use `nohup` to run each sample in the background and proceed with next one. This will in essence, submit each command as a separate job to run on a different core. These will print to separate log files sample.out instead of to the terminal. It will also continue to run, even if you disconnect from SSH or close the terminal.
  
  ```
  for sample in sample1 sample2 sample3
     do
     for lib in lib1 lib2 lib3
       do
       nohup bash script.sh $sample $lib > ${sample}.out &
       done
     done
  
  ```
  
  You should only do this for single-threaded processes. For multi-threaded processes or memory-intensive jobs, this will demand a lot from the machine. Even if this isn’t a shared server, it will perform poorly if you run out of memory and have to “swap” memory by writing to disk. So it’s best to submit them one-by-one (above) or use a cluster with a job scheduler.
  
  
  ---
  
  