Answers.Body: <p>Since there doesn't seem to be an easy way to run this in parallel, you could instead break the job into sections. For example, separate each chromosome into its own gtf file, extract the sequences using that file and then cat them all together.</p>
  
  <p>The commands would be something like this (using the human hg38 genome and the gff GENCODE annotations downloaded from <a href="https://www.gencodegenes.org/releases/current.html" rel="nofollow noreferrer">here</a>):</p>
  
  <p>Split the file into one file per chromosome (or whatever else you have as the first field of your gff/gtf file):</p>
  
  <pre><code>tmp=$(mktemp -d); ## create a temp directory
  awk -vtmp="$tmp" '($1!~/^#/){print &gt; tmp"/"$1".gff"}' gencode.v27.annotation.gtf 
  </code></pre>
  
  <p>Extract the sequences described in each file, running each command in the background (<code>&amp;</code>):</p>
  
  <pre><code>for f in $tmp/*; do 
      gffread -w "$f.fa" -g hg38.fa "$f" &amp; 
  done
  </code></pre>
  
  <p>Concatenate them into one file:</p>
  
  <pre><code>cat "$tmp/*fa" &gt; all.fa
  </code></pre>
  
  <p>You can combine all this into a single command, using <code>wait</code> to make sure you don't concatenate until all of the sub-commands have finished:</p>
  
  <pre><code>tmp=$(mktemp -d); 
  awk -vtmp="$tmp" '($1!~/^#/){print &gt; tmp"/"$1".gff"}' gencode.v27.annotation.gtf &amp;&amp; 
      for f in "$tmp"/*; do 
          gffread-0.9.12.Linux_x86_64/gffread -w $f.fa -g hg38.fa "$f" &amp; 
      done 
      wait 
      cat "$tmp"/*fa &gt; all.fa
  </code></pre>
  
  <p>Or, just save it as a script called pgffread.sh (or whatever):</p>
  
  <pre class="lang-sh prettyprint-override"><code>#!/bin/sh
  gtf="$1"
  genome="$2"
  
  tmp=$(mktemp -d);
  awk -vtmp="$tmp" '($1!~/^#/){print &gt; tmp"/"$1".gff"}' "$gtf" &amp;&amp;
      for f in "$tmp"/*; do
        gffread -w "$f".fa -g "$genome" "$f" &amp;
      done
  wait
  cat "$tmp"/*fa 
  rm -rf "$tmp"
  </code></pre>
  
  <p>And then run it giving the genome and annotation files as parameters:</p>
  
  <pre><code>pgffread.sh gencode.v27.annotation.gtf hg38.fa &gt; all.fa
  </code></pre>
  
  <p>On the system I ran it on, which has 48 cores, the time difference was significant (although it's so fast anyway, I really don't know if it is worth the effort unless you do this very often):</p>
  
  <pre><code>$ time gffread -w out.fa -g hg38.fa gencode.v27.annotation.gtf
  real    1m52.072s
  user    1m3.160s
  sys     0m47.257s
  
  $ time ./pgffread.sh gencode.v27.annotation.gtf hg38.fa &gt; all.fa
  real    0m17.533s
  user    1m5.228s
  sys     0m43.039s
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-03-15T07:17:50.203Z
Answers.Text: thanks! yes, I should have thought of breaking the reference into smaller bits.
Answers.UserDisplayName: aechchiki
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-03-14T23:30:11.627Z
Answers.LastActivityDate: 2018-03-14T23:30:11.627Z
Answers.UserDisplayName: terdon
Body: <p>Is there any possibility to run <a href="http://ccb.jhu.edu/software/stringtie/gff.shtml" rel="noreferrer">gffread</a> in multi-thread mode? The answer seems to be 'no' from the <a href="http://manpages.ubuntu.com/manpages/trusty/man1/gffread.1.html" rel="noreferrer">manual</a> (or <code>gffread -h</code>), as no multi-thread option is mentioned.</p>
  
  <p>I'm mostly using this utility to extract transcript sequences (FASTA) from annotation files (GTF). In single-thread mode, runtime is acceptable in most cases, but would like to speed up a bit in some cases (especially for big genomes).</p>
  
  <p>Note that I already tried, as mentioned in the manual, to index the genome first (which, again, cuts the runtime in most but not all cases): </p>
  
  <blockquote>
    <p>Note that the retrieval of the transcript sequences this way is going to be much faster if a fasta index file (genome.fa.fai in this example) is found in the same directory with the genomic fasta file. </p>
    
    <p>Such an index file can be created with the samtools utility prior to running gffread, like this: <code>samtools faidx genome.fa</code></p>
    
    <p>Then in subsequent runs using the -g option gffread will find this
    FASTA index and use it to speed up the extraction of transcript
    sequences.</p>
  </blockquote>
  
  <p>I'm also open to different alternatives than <code>gffread</code> to convert GTF -> FASTA. </p>
  
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2018-03-14T16:43:58.860Z
Comments.Text: I have used gffread a few times. My impression is it is fast. As I have just tried, for Ensembl human annotations, it gets all transcripts in 29 sec. This should be faster than many other operations on human genome. How often do you want to run gffread?
Comments.UserDisplayName: user172818
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2018-03-14T21:40:52.613Z
Comments.Text: Thank you. Yes it is usually quite fast. Actually I was adressing this question since it took me more than 8min to get the transcripts out of a putative genome size of 1.4Gb, but could not reproduce the slow runtime on another server (where it took me around 110sec). Weird.
Comments.UserDisplayName: aechchiki
Tags: <rna-seq><fasta><file-formats><gtf>
Title: Run gffread in multi-thread mode
ThreadText: # Run gffread in multi-thread mode
  
  (Asked by: username_0 on 2018-03-14)
  
  Is there any possibility to run [gffread](http://ccb.jhu.edu/software/stringtie/gff.shtml) in multi-thread mode? The answer seems to be 'no' from the [manual](http://manpages.ubuntu.com/manpages/trusty/man1/gffread.1.html) (or `gffread -h`), as no multi-thread option is mentioned.
  
  I'm mostly using this utility to extract transcript sequences (FASTA) from annotation files (GTF). In single-thread mode, runtime is acceptable in most cases, but would like to speed up a bit in some cases (especially for big genomes).
  
  Note that I already tried, as mentioned in the manual, to index the genome first (which, again, cuts the runtime in most but not all cases):
  
  > Note that the retrieval of the transcript sequences this way is going to be much faster if a fasta index file (genome.fa.fai in this example) is found in the same directory with the genomic fasta file.
  >
  > Such an index file can be created with the samtools utility prior to running gffread, like this: `samtools faidx genome.fa`
  >
  > Then in subsequent runs using the -g option gffread will find this
  > FASTA index and use it to speed up the extraction of transcript
  > sequences.
  
  I'm also open to different alternatives than `gffread` to convert GTF -> FASTA.
  
  **Tags:** rna-seq, fasta, file-formats, gtf
  
  > **Comments:**
  > **username_1**: I have used gffread a few times. My impression is it is fast. As I have just tried, for Ensembl human annotations, it gets all transcripts in 29 sec. This should be faster than many other operations on human genome. How often do you want to run gffread?  
  > **username_0**: Thank you. Yes it is usually quite fast. Actually I was adressing this question since it took me more than 8min to get the transcripts out of a putative genome size of 1.4Gb, but could not reproduce the slow runtime on another server (where it took me around 110sec). Weird.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2018-03-14. Score: 3 | Accepted answer
  
  Since there doesn't seem to be an easy way to run this in parallel, you could instead break the job into sections. For example, separate each chromosome into its own gtf file, extract the sequences using that file and then cat them all together.
  
  The commands would be something like this (using the human hg38 genome and the gff GENCODE annotations downloaded from [here](https://www.gencodegenes.org/releases/current.html)):
  
  Split the file into one file per chromosome (or whatever else you have as the first field of your gff/gtf file):
  
  ```
  tmp=$(mktemp -d); ## create a temp directory
  awk -vtmp="$tmp" '($1!~/^#/){print > tmp"/"$1".gff"}' gencode.v27.annotation.gtf 
  
  ```
  
  Extract the sequences described in each file, running each command in the background (`&`):
  
  ```
  for f in $tmp/*; do 
      gffread -w "$f.fa" -g hg38.fa "$f" & 
  done
  
  ```
  
  Concatenate them into one file:
  
  ```
  cat "$tmp/*fa" > all.fa
  
  ```
  
  You can combine all this into a single command, using `wait` to make sure you don't concatenate until all of the sub-commands have finished:
  
  ```
  tmp=$(mktemp -d); 
  awk -vtmp="$tmp" '($1!~/^#/){print > tmp"/"$1".gff"}' gencode.v27.annotation.gtf && 
      for f in "$tmp"/*; do 
          gffread-0.9.12.Linux_x86_64/gffread -w $f.fa -g hg38.fa "$f" & 
      done 
      wait 
      cat "$tmp"/*fa > all.fa
  
  ```
  
  Or, just save it as a script called pgffread.sh (or whatever):
  
  ```sh
  #!/bin/sh
  gtf="$1"
  genome="$2"
  
  tmp=$(mktemp -d);
  awk -vtmp="$tmp" '($1!~/^#/){print > tmp"/"$1".gff"}' "$gtf" &&
      for f in "$tmp"/*; do
        gffread -w "$f".fa -g "$genome" "$f" &
      done
  wait
  cat "$tmp"/*fa 
  rm -rf "$tmp"
  
  ```
  
  And then run it giving the genome and annotation files as parameters:
  
  ```
  pgffread.sh gencode.v27.annotation.gtf hg38.fa > all.fa
  
  ```
  
  On the system I ran it on, which has 48 cores, the time difference was significant (although it's so fast anyway, I really don't know if it is worth the effort unless you do this very often):
  
  ```
  $ time gffread -w out.fa -g hg38.fa gencode.v27.annotation.gtf
  real    1m52.072s
  user    1m3.160s
  sys     0m47.257s
  
  $ time ./pgffread.sh gencode.v27.annotation.gtf hg38.fa > all.fa
  real    0m17.533s
  user    1m5.228s
  sys     0m43.039s
  
  ```
  
  > **Comments:**
  > **username_0**: thanks! yes, I should have thought of breaking the reference into smaller bits.  
  
  ---
  
  