Answers.Body: <p>You will probably be interested in the following UCSC wiki page, which explains how to go from most of the UCSC tables to GTF/GFF:<br>
  <a href="http://genomewiki.ucsc.edu/index.php/Genes_in_gtf_or_gff_format" rel="noreferrer">http://genomewiki.ucsc.edu/index.php/Genes_in_gtf_or_gff_format</a></p>
  
  <p>The basic gist is that UCSC doesn't store any data internally as GTF or GFF, and so you will need to use our genePredToGtf utility in order to convert from our internally stored genePred format into GTF/GFF3.</p>
  
  <p>This is the command I usually give when we get this question on our mailing list:</p>
  
  <pre>
  $ mysql --user=genome --host=genome-mysql.cse.ucsc.edu -A -N -e "select * from refGene" hg19 | cut -f2- | genePredToGtf -source=hg19.refGene.ucsc file stdin stdout
  </pre>
  
  <p>Change stdout to the output filename you want in the last command to get an hg19 refGene GTF file:</p>
  
  <pre>
  chr1    hg19.refGene.ucsc   transcript  11869   14362   .   +   .   gene_id "LOC102725121"; transcript_id "NR_148357";  gene_name "LOC102725121";
  chr1    hg19.refGene.ucsc   exon    11869   12227   .   +   .   gene_id "LOC102725121"; transcript_id "NR_148357"; exon_number "1"; exon_id "NR_148357.1"; gene_name "LOC102725121";
  chr1    hg19.refGene.ucsc   exon    12613   12721   .   +   .   gene_id "LOC102725121"; transcript_id "NR_148357"; exon_number "2"; exon_id "NR_148357.2"; gene_name "LOC102725121";
  chr1    hg19.refGene.ucsc   exon    13221   14362   .   +   .   gene_id "LOC102725121"; transcript_id "NR_148357"; exon_number "3"; exon_id "NR_148357.3"; gene_name "LOC102725121";
  chr1    hg19.refGene.ucsc   transcript  11874   14409   .   +   .   gene_id "DDX11L1"; transcript_id "NR_046018";  gene_name "DDX11L1";
  ...
  </pre>
  
  <p>If you have further questions about UCSC data or tools feel free to send your question to one of the below mailing lists:</p>
  
  <ul>
  <li>General questions: genome@soe.ucsc.edu</li>
  <li>Questions involving private data: genome-www@soe.ucsc.edu</li>
  <li>Questions involving mirror sites: genome-mirror@ose.ucsc.edu</li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-09-21T18:35:17.683Z
Answers.Text: It would be rather nice if the "GTF" format output from the UCSC table browser simply returned exactly this (including the `gene` entries, which are always missing). This is the sort of thing people are actually expecting from it.
Answers.UserDisplayName: Devon Ryan
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-09-21T19:14:01.417Z
Answers.Text: I agree that this would be ideal. The problem is that the gene_id's returned may not be what people want. For example, if you were trying to get a GTF output for the wgEncodeGencodeBasicV27 table, would you want a gene_id like DDX11L1 or a gene_id like ENSG*? The other issue is the plain genePred table format doesn't even have a name2 field, so what do we assign as gene_id in this case? What if the table is some other variation of genePred that also doesn't have a name2? We use many different table variations, and it is hard to support a GTF output for all of them.
Answers.UserDisplayName: Christopher Lee
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-09-22T16:12:48.097Z
Answers.Text: @ChristopherLee, that's perfect, thank you. Regarding your response to @DevonRyan's comment, I can appreciate that it's difficult (and maybe unrealistic) to support all possible variations, but I'd also argue that HUGO is probably what most people want. In either case, I'm certain that reporting the transcript ID for *both* `gene_id` and `transcript_id` is not ideal! :-)
Answers.UserDisplayName: Mark Ebbert
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-09-21T18:11:33.103Z
Answers.LastActivityDate: 2017-09-21T18:11:33.103Z
Answers.UserDisplayName: Christopher Lee
Body: <p>I'm not sure what I'm missing, but I'm struggling to find an official <code>hg38</code> <code>GTF</code> file with RefSeq annotations. I'd like to provide the <code>GTF</code> to <a href="https://combine-lab.github.io/salmon/" rel="nofollow noreferrer">Salmon</a> to get gene-level annotations. </p>
  
  <p>Here's Salmon's help info for <code>--geneMap</code>:</p>
  
  <blockquote>
    <p>File containing a mapping of
                                            transcripts to genes.  If this file is
                                            provided Salmon will output both
                                            quant.sf and quant.genes.sf files,
                                            where the latter contains aggregated
                                            gene-level abundance estimates.  The
                                            transcript to gene mapping should be
                                            provided as either a GTF file, or a in
                                            a simple tab-delimited format where
                                            each line contains the name of a
                                            transcript and the gene to which it
                                            belongs separated by a tab.  The
                                            extension of the file is used to
                                            determine how the file should be
                                            parsed.  Files ending in '.gtf', '.gff'
                                            or '.gff3' are assumed to be in GTF
                                            format; files with any other extension
                                            are assumed to be in the simple format.
                                            In GTF / GFF format, the
                                            "transcript_id" is assumed to contain
                                            the transcript identifier and the
                                            "gene_id" is assumed to contain the
                                            corresponding gene identifier.</p>
  </blockquote>
  
  <p>Here's what I've tried:</p>
  
  <ol>
  <li>NCBI seems to only provide <code>GFF</code> files. I know you can convert between the two, but that hasn't worked well for me in the past. Both file formats allow a lot of freedom, which makes conversions sloppy. Salmon, specifically, is looking for the <code>gene_id</code> and <code>transcript_id</code> annotations.</li>
  <li>UCSC provides <code>GTF</code> files from RefSeq, but the <code>gene_id</code> annotation is identical to the <code>transcript_id</code> annotation (i.e., it's the NM number). Or maybe there's an option I'm missing.</li>
  <li>Ensembl provides exactly the format and all of the information I could hope for, except the <code>transcript_id</code> is the Ensembl ID (ENST#), not RefSeq (NM#).</li>
  </ol>
  
  <p>Can anyone tell me how to get a <code>GTF</code> file with hg38 RefSeq annotations?</p>
  
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-09-21T07:17:31.500Z
Comments.Text: refseq IDs are per-transcript, so it's expected that the gene and transcript IDs will be the same. Given that, what would you like as the gene_id?
Comments.UserDisplayName: Devon Ryan
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-09-21T14:25:10.060Z
Comments.Text: @Devon Ryan I'm guessing the EntrezGene ID or HUGO symbol?
Comments.UserDisplayName: Ian Sudbery
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-09-21T14:29:50.780Z
Comments.Text: @Mark Ebbert: What are your down stream applications? The as far as I'm aware, the salmon output doesn't include the gene_id, so it shouldn't matter that it is the same as the transcript_id.
Comments.UserDisplayName: Ian Sudbery
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-09-21T15:19:55.237Z
Comments.Text: I appreciate your help. I added Salmon's help on `--geneMap`. They expect `gene_id` to contain a "gene identifier". I understand that could be several things, but I would prefer the HUGO symbol, as @IanSudbery suggested. In my perusing, I thing Salmon should probably target the `gene_name`, at least how Ensembl formats the GTF. In either case, I'm surprised by @DevonRyan's comment that the `gene_id` would equal the `transcript_id`. Not arguing that's the case, but what good does that do anyone?? :-)
Comments.UserDisplayName: Mark Ebbert
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-09-21T15:26:28.733Z
Comments.Text: @IanSudbery, I ultimately want to aggregate transcript expression into the parent genes. I found last night that one of the Salmon authors now recommend `tximport` (https://github.com/COMBINE-lab/salmon/issues/98), which I will use. But this isn't the first time I've "needed" a RefSeq GTF and couldn't find one. Just surprises me, but maybe that indicates I need to assess my approach?
Comments.UserDisplayName: Mark Ebbert
Tags: <ngs><public-databases><sequence-annotation><human-genome><gtf>
Title: hg38 GTF file with RefSeq annotations
ThreadText: # hg38 GTF file with RefSeq annotations
  
  (Asked by: username_0 on 2017-09-21)
  
  I'm not sure what I'm missing, but I'm struggling to find an official `hg38` `GTF` file with RefSeq annotations. I'd like to provide the `GTF` to [Salmon](https://combine-lab.github.io/salmon/) to get gene-level annotations.
  
  Here's Salmon's help info for `--geneMap`:
  
  > File containing a mapping of
  > transcripts to genes. If this file is
  > provided Salmon will output both
  > quant.sf and quant.genes.sf files,
  > where the latter contains aggregated
  > gene-level abundance estimates. The
  > transcript to gene mapping should be
  > provided as either a GTF file, or a in
  > a simple tab-delimited format where
  > each line contains the name of a
  > transcript and the gene to which it
  > belongs separated by a tab. The
  > extension of the file is used to
  > determine how the file should be
  > parsed. Files ending in '.gtf', '.gff'
  > or '.gff3' are assumed to be in GTF
  > format; files with any other extension
  > are assumed to be in the simple format.
  > In GTF / GFF format, the
  > "transcript\_id" is assumed to contain
  > the transcript identifier and the
  > "gene\_id" is assumed to contain the
  > corresponding gene identifier.
  
  Here's what I've tried:
  
  1. NCBI seems to only provide `GFF` files. I know you can convert between the two, but that hasn't worked well for me in the past. Both file formats allow a lot of freedom, which makes conversions sloppy. Salmon, specifically, is looking for the `gene_id` and `transcript_id` annotations.
  2. UCSC provides `GTF` files from RefSeq, but the `gene_id` annotation is identical to the `transcript_id` annotation (i.e., it's the NM number). Or maybe there's an option I'm missing.
  3. Ensembl provides exactly the format and all of the information I could hope for, except the `transcript_id` is the Ensembl ID (ENST#), not RefSeq (NM#).
  
  Can anyone tell me how to get a `GTF` file with hg38 RefSeq annotations?
  
  **Tags:** ngs, public-databases, sequence-annotation, human-genome, gtf
  
  > **Comments:**
  > **username_1**: refseq IDs are per-transcript, so it's expected that the gene and transcript IDs will be the same. Given that, what would you like as the gene_id?  
  > **username_2**: @username_1 I'm guessing the EntrezGene ID or HUGO symbol?  
  > **username_2**: @username_0: What are your down stream applications? The as far as I'm aware, the salmon output doesn't include the gene_id, so it shouldn't matter that it is the same as the transcript_id.  
  > **username_0**: I appreciate your help. I added Salmon's help on `--geneMap`. They expect `gene_id` to contain a "gene identifier". I understand that could be several things, but I would prefer the HUGO symbol, as @username_2 suggested. In my perusing, I thing Salmon should probably target the `gene_name`, at least how Ensembl formats the GTF. In either case, I'm surprised by @username_1's comment that the `gene_id` would equal the `transcript_id`. Not arguing that's the case, but what good does that do anyone?? :-)  
  > **username_0**: @username_2, I ultimately want to aggregate transcript expression into the parent genes. I found last night that one of the Salmon authors now recommend `tximport` (https://github.com/COMBINE-lab/salmon/issues/98), which I will use. But this isn't the first time I've "needed" a RefSeq GTF and couldn't find one. Just surprises me, but maybe that indicates I need to assess my approach?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_3 on 2017-09-21. Score: 5 | Accepted answer
  
  You will probably be interested in the following UCSC wiki page, which explains how to go from most of the UCSC tables to GTF/GFF:  
  <http://genomewiki.ucsc.edu/index.php/Genes_in_gtf_or_gff_format>
  
  The basic gist is that UCSC doesn't store any data internally as GTF or GFF, and so you will need to use our genePredToGtf utility in order to convert from our internally stored genePred format into GTF/GFF3.
  
  This is the command I usually give when we get this question on our mailing list:
  
  ```
  
  $ mysql --user=genome --host=genome-mysql.cse.ucsc.edu -A -N -e "select * from refGene" hg19 | cut -f2- | genePredToGtf -source=hg19.refGene.ucsc file stdin stdout
  
  ```
  
  Change stdout to the output filename you want in the last command to get an hg19 refGene GTF file:
  
  ```
  
  chr1    hg19.refGene.ucsc   transcript  11869   14362   .   +   .   gene_id "LOC102725121"; transcript_id "NR_148357";  gene_name "LOC102725121";
  chr1    hg19.refGene.ucsc   exon    11869   12227   .   +   .   gene_id "LOC102725121"; transcript_id "NR_148357"; exon_number "1"; exon_id "NR_148357.1"; gene_name "LOC102725121";
  chr1    hg19.refGene.ucsc   exon    12613   12721   .   +   .   gene_id "LOC102725121"; transcript_id "NR_148357"; exon_number "2"; exon_id "NR_148357.2"; gene_name "LOC102725121";
  chr1    hg19.refGene.ucsc   exon    13221   14362   .   +   .   gene_id "LOC102725121"; transcript_id "NR_148357"; exon_number "3"; exon_id "NR_148357.3"; gene_name "LOC102725121";
  chr1    hg19.refGene.ucsc   transcript  11874   14409   .   +   .   gene_id "DDX11L1"; transcript_id "NR_046018";  gene_name "DDX11L1";
  ...
  
  ```
  
  If you have further questions about UCSC data or tools feel free to send your question to one of the below mailing lists:
  
  * General questions: genome@soe.ucsc.edu
  * Questions involving private data: genome-www@soe.ucsc.edu
  * Questions involving mirror sites: genome-mirror@ose.ucsc.edu
  
  > **Comments:**
  > **username_1**: It would be rather nice if the "GTF" format output from the UCSC table browser simply returned exactly this (including the `gene` entries, which are always missing). This is the sort of thing people are actually expecting from it.  
  > **username_3**: I agree that this would be ideal. The problem is that the gene_id's returned may not be what people want. For example, if you were trying to get a GTF output for the wgEncodeGencodeBasicV27 table, would you want a gene_id like DDX11L1 or a gene_id like ENSG*? The other issue is the plain genePred table format doesn't even have a name2 field, so what do we assign as gene_id in this case? What if the table is some other variation of genePred that also doesn't have a name2? We use many different table variations, and it is hard to support a GTF output for all of them.  
  > **username_0**: @username_3, that's perfect, thank you. Regarding your response to @username_1's comment, I can appreciate that it's difficult (and maybe unrealistic) to support all possible variations, but I'd also argue that HUGO is probably what most people want. In either case, I'm certain that reporting the transcript ID for *both* `gene_id` and `transcript_id` is not ideal! :-)  
  
  ---
  
  