Answers.Body: <p>For generating the index, you should be using one of <code>-i</code> or <code>--index</code>, but not both:</p>
  
  <pre><code>./kallisto index --make-unique -i index.fa.idx index.fa 
  </code></pre>
  
  <p>You need to point kallisto to the generated index file (usually has a <code>.idx</code> extension), rather than the fasta file:</p>
  
  <pre><code>for file in *.fasta;
    do kallisto quant -i index.fa.idx --single -l 200 -s 0.1 -o "${file}-aligned" "${file}";
  done
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-06-30T09:30:44.477Z
Answers.Text: Why run it sequentially? Shouldn't this be parallelized, i.e. multiprocessed
Answers.UserDisplayName: Chris_Rands
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-06-30T11:38:47.110Z
Answers.Text: You're asking the wrong person; try adding that as a comment to the question. `Kallisto quant` allows multi-threaded operation with `-t <threads>`.
Answers.UserDisplayName: gringer
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-06-30T15:18:44.913Z
Answers.Text: I get the same error when I point to the index file
  
      Error: incompatible indices. Found version 0, expected version 10
Answers.UserDisplayName: ZincFingers
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-06-30T16:05:52.333Z
Answers.Text: maybe it's the index command that needs a bit of attention... I'll update the commands to include index generation
Answers.UserDisplayName: gringer
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-06-30T16:20:11.787Z
Answers.Text: I managed to get this to work after fiddling with the index generation:
  ' for f in *.fasta' do kallisto quant -i index --single -l 200 -s 0.1 -o "${file}-aligned" "${file)"; '
  
  But it appears that it's repeatedly analyzing one of the 10 .fasta files I placed in the directory (also I'm not sure how to get that line to appear as a code)
Answers.UserDisplayName: ZincFingers
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-06-30T16:24:38.760Z
Answers.Text: @gringer - just an additional comment on multithreading: as of version `0.43.1`, the `-t <threads>` parameter doesn't work should you wish to look at the alignments (which `kallisto quant` can output upon supplying `--pseudobam`. I usually get around it by running several individual `kallisto quant` commands concurrently.
Answers.UserDisplayName: flatley176
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-06-30T21:12:51.717Z
Answers.Text: @gringer Using `-t` is fine but it’s actually almost certainly much more efficient to process the *files* in parallel rather than in a sequential loop.
Answers.UserDisplayName: Konrad Rudolph
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-07-03T09:10:23.250Z
Answers.Text: `-t` is not going to help, it's only useful if you've enabled bootstrapping, otherwise you can set it as high as you want but kallisto will still run single core.  Multi-threaded execution is used only for concurrently running the bootstrapping.
Answers.UserDisplayName: Matt Bashton
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-07-03T09:11:50.737Z
Answers.Text: If you're using tximport for downstream analysis in DESeq2 rather than Sleuth the bootstrapping won't be used as it's not taking into account quantification uncertainty via the bootstrapping procedure, so you need to consider if this is needed.
Answers.UserDisplayName: Matt Bashton
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-06-29T23:43:58.127Z
Answers.LastActivityDate: 2017-06-30T16:13:19.153Z
Answers.LastEditDate: 2017-06-30T16:13:19.153Z
Answers.UserDisplayName: gringer
Answers.Body: <p><strong>Running the jobs in parallel</strong></p>
  
  <p>Iteratively is one solution, but since you're not using bootstrapping, which can be run multithread (and is not needed if you're taking the popular <a href="http://bioconductor.org/packages/release/bioc/vignettes/tximport/inst/doc/tximport.html" rel="nofollow noreferrer">tximport</a> route) you can do better using <a href="https://www.gnu.org/software/parallel/" rel="nofollow noreferrer">GNU parallel</a>, this will enable you to run as many jobs as you have execution threads on your system simultaneously.  Note that Kalliso only runs multithreaded with <code>-t</code> if you're using bootstrapping.</p>
  
  <ol>
  <li><p>You'll need to make the index first:</p>
  
  <pre><code>kallisto index --make-unique -I index.fa.idx index.fa
  </code></pre></li>
  <li><p>Next lets define the input FASTQ files and the sample IDs you want to use using a tab-delimited input file like so:</p>
  
  <pre><code>sample_1    sample_1_R1.fastq.gz
  sample_2    sample_2_R1.fastq.gz
  sample_3    sample_3_R1.fastq.gz
  </code></pre>
  
  <p>Here the first column is the sample ID which will be used for output and the second separated by a tab character is the input file. </p>
  
  <p>You can produce this using a text editor but as a short-cut you can feed the list of fastq.gz you have into a list via:</p>
  
  <pre><code>ls -1 *.fastq.gz &gt; editme
  </code></pre></li>
  <li><p>Next we create a list of jobs to run on our system using a utility script <code>Make_job_list_kallisto.sh</code> which consumes our tab delimited input file list above.  I've used your <code>-l</code> and <code>-s</code> parameters below, note that with paired end data normally these are estimated for you.</p>
  
  <pre><code>#!/bin/bash -eu
  
  [ $# -ne 2 ] &amp;&amp; { echo -en "\n*** This script generates jobs for GNU parallel. *** \n\n Error Nothing to do, usage: &lt; input tab delimited list &gt; &lt; output run list file &gt;\n\n" ; exit 1; }
  set -o pipefail
  
  # Get command-line args
  INPUT_LIST=$1
  OUTPUT=$2
  
  # Set counter
  COUNT=1
  END=$(wc -l $INPUT_LIST | awk '{print $1}')
  
  echo " "
  echo " * Input file is: $INPUT_LIST"
  echo " * Number of runs: $END"
  echo " * Output job list for GNU parallel saved to: $OUTPUT"
  echo " "
  
  # Main bit of command-line for job
  CMD="kallisto quant -i index.fa.idx --single -l 200 -s 0.1"
  
  # Main Loop
  [ -e $OUTPUT ] &amp;&amp; rm $OUTPUT
  while [ $COUNT -le $END ];
  do
      LINE=( $(awk "NR==$COUNT" $INPUT_LIST) )
      # Make file list
      echo "Working on $COUNT of $END Sample ID: ${LINE[0]}, Files ${LINE[@]:1}"
      echo "$CMD -o ${LINE[0]} ${LINE[@]:1}" &gt;&gt; $OUTPUT
      ((COUNT++))
  done
  </code></pre>
  
  <p>We use the script like so:</p>
  
  <pre><code> Make_job_list_kallisto.sh input_file_list.txt job_list.txt
  </code></pre></li>
  <li><p>Now we have a job list for GNU parallel we can run our jobs in parallel like so:</p>
  
  <pre><code>parallel --progress --jobs 4 --joblog kallisto_joblog.txt &lt; job_list.txt
  </code></pre>
  
  <p>This will run 4 of the kallisto jobs simultaneously (Increase this if you have more threads - don't mind consuming all the IO/CPU threads on your system) producing a nice progress report of those completed, additionally a job log will be written to the file <code>kallisto_joblog.txt</code>.  </p></li>
  </ol>
  
  <p>For paired end data I already have these scripts + similar instructions on <a href="https://github.com/MattBashton/grolar" rel="nofollow noreferrer">GitHub</a> which people might find useful. </p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-07-03T20:49:52.250Z
Answers.Text: @ZincFingers, so `-I` upper case for specifying the index in my run generation scrip should in fact be lower case `-i`.  I've tested this and it works for me.  This might be your issue or you might have some other error.  If it's still not working try pasting in a few lines of your jobs list here.
Answers.UserDisplayName: Matt Bashton
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-07-03T20:56:04.290Z
Answers.Text: The error I was getting was my fault, I fixed it. Now, when I run the parallel command, I get an error from kallisto saying 
  
  Error: missing read files
  Error: file srr3884878.gz exists and is not a directory
  
  It says this for all 17 of my files
Answers.UserDisplayName: ZincFingers
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-07-03T21:35:21.343Z
Answers.Text: I suspect you still have an issue with you input file to `Make_job_list_kallisto.sh` or the jobs it's generating, without seeing the input sample ID file list or the job list it's very difficult to debug.
Answers.UserDisplayName: Matt Bashton
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-07-03T21:39:32.763Z
Answers.Text: Check the diagnostic output of `Make_job_list_kallisto.sh` is sane, I get:
  `* Input file is: test_for_BS.txt`
  `* Number of runs: 4`
  `* Output for GNU parallel saved to: test_job_list.txt`
  
  `Working on 1 of 4 Sample ID: test1, Files ATCACG_L004_R1_001.fastq.gz`
  `Working on 2 of 4 Sample ID: test2, Files GTCCGC_L004_R1_001.trimmed.fastq.gz`
  
  etc.
Answers.UserDisplayName: Matt Bashton
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-07-03T21:54:28.363Z
Answers.Text: I think you might have forgotten to give an output dir or sample ID in the first column of your tab delimited input list you're giving to `Make_job_list_kallisto.sh` because it looks like kallisto is missing the variable which would be supplied to `-o` and treating the input fastq.gz file as the output dir, it's then not finding the gz.fastq file as it's trying to use that as the output dir.
Answers.UserDisplayName: Matt Bashton
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-07-03T22:09:48.640Z
Answers.Text: You were right! I fixed the input file and it seems to have resolved the issue. Many thanks for your fantastic solution and bearing with me while I trudged through this.
Answers.UserDisplayName: ZincFingers
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-07-03T22:25:54.190Z
Answers.Text: Glad you got it working!
Answers.UserDisplayName: Matt Bashton
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-07-03T09:02:32.610Z
Answers.LastActivityDate: 2017-07-03T20:50:27.253Z
Answers.LastEditDate: 2017-07-03T20:50:27.253Z
Answers.UserDisplayName: Matt Bashton
Body: <p>I am on a Mac using UNIX. I am trying to use the kallisto quant command on all files in a directory (instead of manually entering them). Because I'm running the analysis against the same index file, I first enter the following:</p>
  
  <pre><code>./kallisto index -i --index --make-unique index.fa 
  </code></pre>
  
  <p>This successfully creates an index file. Then, I tried this:</p>
  
  <pre><code>for file in *.fasta; do kallisto quant --single -l 200 -s 0.1 -o $file-aligned 
  "$file"; done
  </code></pre>
  
  <p>While the $file-aligned folders are created, they're empty. I get an error that states that the index file is missing. So I assumed I would need to specify the index file in the command line. I then tried this:</p>
  
  <pre><code>for file in *.fasta; do kallisto quant -i index.fa --single -l 200 -s 0.1 -o
  $file-aligned  "$file"; done
  </code></pre>
  
  <p>I get an error that says "Error: incompatible indices. Found version 0, expected version 10. </p>
  
  <p>I have now also tried the following:</p>
  
  <pre><code>for file in *.fasta; do ./kallisto quant -i index --single - 200 -s 0.01 -o 
  "${file}-aligned" "${file}"; 
  </code></pre>
  
  <p>but I get an error saying the index file is not found.</p>
  
  <hr>
  
  <p>I am now running </p>
  
  <pre><code>or file in *.fasta; do ./kallisto quant -i index --single -l 200 -s 0.1 -o $file-aligned 
  </code></pre>
  
  <p>"$file"; done</p>
  
  <p>This sort of works. The index file error messages are gone, but it runs quant on the same file 10 times instead of each file once. If I run echo $file it spits back the file that my command is running repeatedly. </p>
  
  <p>If I close terminal, start a new session and run the same command, I get</p>
  
  <pre><code>Error: file not found ""
  </code></pre>
  
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-06-29T23:44:49.577Z
Comments.Text: Why are you using fasta files for reads? Kallisto expects FASTQ files as input.
Comments.UserDisplayName: gringer
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-06-30T07:55:48.413Z
Comments.Text: Does kallisto use quality scores of reads? I had an impression that it does not, but I don't have a reference to backup it....
Comments.UserDisplayName: Kamil S Jaron
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-05-12T18:15:09.330Z
Comments.Text: I down-voted because this post does not pose a question.
Comments.UserDisplayName: winni2k
Tags: <rna-seq>
Title: Run kallisto iteratively across many samples
ThreadText: # Run kallisto iteratively across many samples
  
  (Asked by: username_0 on 2017-06-29)
  
  I am on a Mac using UNIX. I am trying to use the kallisto quant command on all files in a directory (instead of manually entering them). Because I'm running the analysis against the same index file, I first enter the following:
  
  ```
  ./kallisto index -i --index --make-unique index.fa 
  
  ```
  
  This successfully creates an index file. Then, I tried this:
  
  ```
  for file in *.fasta; do kallisto quant --single -l 200 -s 0.1 -o $file-aligned 
  "$file"; done
  
  ```
  
  While the $file-aligned folders are created, they're empty. I get an error that states that the index file is missing. So I assumed I would need to specify the index file in the command line. I then tried this:
  
  ```
  for file in *.fasta; do kallisto quant -i index.fa --single -l 200 -s 0.1 -o
  $file-aligned  "$file"; done
  
  ```
  
  I get an error that says "Error: incompatible indices. Found version 0, expected version 10.
  
  I have now also tried the following:
  
  ```
  for file in *.fasta; do ./kallisto quant -i index --single - 200 -s 0.01 -o 
  "${file}-aligned" "${file}"; 
  
  ```
  
  but I get an error saying the index file is not found.
  
  ---
  
  I am now running
  
  ```
  or file in *.fasta; do ./kallisto quant -i index --single -l 200 -s 0.1 -o $file-aligned 
  
  ```
  
  "$file"; done
  
  This sort of works. The index file error messages are gone, but it runs quant on the same file 10 times instead of each file once. If I run echo $file it spits back the file that my command is running repeatedly.
  
  If I close terminal, start a new session and run the same command, I get
  
  ```
  Error: file not found ""
  
  ```
  
  **Tags:** rna-seq
  
  > **Comments:**
  > **username_1**: Why are you using fasta files for reads? Kallisto expects FASTQ files as input.  
  > **username_2**: Does kallisto use quality scores of reads? I had an impression that it does not, but I don't have a reference to backup it....  
  > **username_3**: I down-voted because this post does not pose a question.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_7 on 2017-07-03. Score: 7 | Accepted answer
  
  **Running the jobs in parallel**
  
  Iteratively is one solution, but since you're not using bootstrapping, which can be run multithread (and is not needed if you're taking the popular [tximport](http://bioconductor.org/packages/release/bioc/vignettes/tximport/inst/doc/tximport.html) route) you can do better using [GNU parallel](https://www.gnu.org/software/parallel/), this will enable you to run as many jobs as you have execution threads on your system simultaneously. Note that Kalliso only runs multithreaded with `-t` if you're using bootstrapping.
  
  1. You'll need to make the index first:
  
     ```
     kallisto index --make-unique -I index.fa.idx index.fa
  
     ```
  2. Next lets define the input FASTQ files and the sample IDs you want to use using a tab-delimited input file like so:
  
     ```
     sample_1    sample_1_R1.fastq.gz
     sample_2    sample_2_R1.fastq.gz
     sample_3    sample_3_R1.fastq.gz
  
     ```
  
     Here the first column is the sample ID which will be used for output and the second separated by a tab character is the input file.
  
     You can produce this using a text editor but as a short-cut you can feed the list of fastq.gz you have into a list via:
  
     ```
     ls -1 *.fastq.gz > editme
  
     ```
  3. Next we create a list of jobs to run on our system using a utility script `Make_job_list_kallisto.sh` which consumes our tab delimited input file list above. I've used your `-l` and `-s` parameters below, note that with paired end data normally these are estimated for you.
  
     ```
     #!/bin/bash -eu
  
     [ $# -ne 2 ] && { echo -en "\n*** This script generates jobs for GNU parallel. *** \n\n Error Nothing to do, usage: < input tab delimited list > < output run list file >\n\n" ; exit 1; }
     set -o pipefail
  
     # Get command-line args
     INPUT_LIST=$1
     OUTPUT=$2
  
     # Set counter
     COUNT=1
     END=$(wc -l $INPUT_LIST | awk '{print $1}')
  
     echo " "
     echo " * Input file is: $INPUT_LIST"
     echo " * Number of runs: $END"
     echo " * Output job list for GNU parallel saved to: $OUTPUT"
     echo " "
  
     # Main bit of command-line for job
     CMD="kallisto quant -i index.fa.idx --single -l 200 -s 0.1"
  
     # Main Loop
     [ -e $OUTPUT ] && rm $OUTPUT
     while [ $COUNT -le $END ];
     do
         LINE=( $(awk "NR==$COUNT" $INPUT_LIST) )
         # Make file list
         echo "Working on $COUNT of $END Sample ID: ${LINE[0]}, Files ${LINE[@]:1}"
         echo "$CMD -o ${LINE[0]} ${LINE[@]:1}" >> $OUTPUT
         ((COUNT++))
     done
  
     ```
  
     We use the script like so:
  
     ```
      Make_job_list_kallisto.sh input_file_list.txt job_list.txt
  
     ```
  4. Now we have a job list for GNU parallel we can run our jobs in parallel like so:
  
     ```
     parallel --progress --jobs 4 --joblog kallisto_joblog.txt < job_list.txt
  
     ```
  
     This will run 4 of the kallisto jobs simultaneously (Increase this if you have more threads - don't mind consuming all the IO/CPU threads on your system) producing a nice progress report of those completed, additionally a job log will be written to the file `kallisto_joblog.txt`.
  
  For paired end data I already have these scripts + similar instructions on [GitHub](https://github.com/username_7/grolar) which people might find useful.
  
  > **Comments:**
  > **username_7**: @username_0, so `-I` upper case for specifying the index in my run generation scrip should in fact be lower case `-i`.  I've tested this and it works for me.  This might be your issue or you might have some other error.  If it's still not working try pasting in a few lines of your jobs list here.  
  > **username_0**: The error I was getting was my fault, I fixed it. Now, when I run the parallel command, I get an error from kallisto saying   Error: missing read files Error: file srr3884878.gz exists and is not a directory  It says this for all 17 of my files  
  > **username_7**: I suspect you still have an issue with you input file to `Make_job_list_kallisto.sh` or the jobs it's generating, without seeing the input sample ID file list or the job list it's very difficult to debug.  
  > **username_7**: Check the diagnostic output of `Make_job_list_kallisto.sh` is sane, I get: `* Input file is: test_for_BS.txt` `* Number of runs: 4` `* Output for GNU parallel saved to: test_job_list.txt`  `Working on 1 of 4 Sample ID: test1, Files ATCACG_L004_R1_001.fastq.gz` `Working on 2 of 4 Sample ID: test2, Files GTCCGC_L004_R1_001.trimmed.fastq.gz`  etc.  
  > **username_7**: I think you might have forgotten to give an output dir or sample ID in the first column of your tab delimited input list you're giving to `Make_job_list_kallisto.sh` because it looks like kallisto is missing the variable which would be supplied to `-o` and treating the input fastq.gz file as the output dir, it's then not finding the gz.fastq file as it's trying to use that as the output dir.  
  > **username_0**: You were right! I fixed the input file and it seems to have resolved the issue. Many thanks for your fantastic solution and bearing with me while I trudged through this.  
  > **username_7**: Glad you got it working!  
  
  ---
  
  ### Answer by username_1 on 2017-06-29. Score: 5
  
  For generating the index, you should be using one of `-i` or `--index`, but not both:
  
  ```
  ./kallisto index --make-unique -i index.fa.idx index.fa 
  
  ```
  
  You need to point kallisto to the generated index file (usually has a `.idx` extension), rather than the fasta file:
  
  ```
  for file in *.fasta;
    do kallisto quant -i index.fa.idx --single -l 200 -s 0.1 -o "${file}-aligned" "${file}";
  done
  
  ```
  
  > **Comments:**
  > **username_4**: Why run it sequentially? Shouldn't this be parallelized, i.e. multiprocessed  
  > **username_1**: You're asking the wrong person; try adding that as a comment to the question. `Kallisto quant` allows multi-threaded operation with `-t <threads>`.  
  > **username_0**: I get the same error when I point to the index file      Error: incompatible indices. Found version 0, expected version 10  
  > **username_1**: maybe it's the index command that needs a bit of attention... I'll update the commands to include index generation  
  > **username_0**: I managed to get this to work after fiddling with the index generation: ' for f in *.fasta' do kallisto quant -i index --single -l 200 -s 0.1 -o "${file}-aligned" "${file)"; '  But it appears that it's repeatedly analyzing one of the 10 .fasta files I placed in the directory (also I'm not sure how to get that line to appear as a code)  
  > **username_5**: @username_1 - just an additional comment on multithreading: as of version `0.43.1`, the `-t <threads>` parameter doesn't work should you wish to look at the alignments (which `kallisto quant` can output upon supplying `--pseudobam`. I usually get around it by running several individual `kallisto quant` commands concurrently.  
  > **username_6**: @username_1 Using `-t` is fine but it’s actually almost certainly much more efficient to process the *files* in parallel rather than in a sequential loop.  
  > **username_7**: `-t` is not going to help, it's only useful if you've enabled bootstrapping, otherwise you can set it as high as you want but kallisto will still run single core.  Multi-threaded execution is used only for concurrently running the bootstrapping.  
  > **username_7**: If you're using tximport for downstream analysis in DESeq2 rather than Sleuth the bootstrapping won't be used as it's not taking into account quantification uncertainty via the bootstrapping procedure, so you need to consider if this is needed.  
  
  ---
  
  