Answers.Body: <p>As Pierre Lindenbaum  pointed out in <a href="https://www.biostars.org/p/9602494/" rel="nofollow noreferrer">your duplicate question</a> on Biostars, GATK's function that checks the validity of a GTF feature fails if <code>gene_type</code> is missing (<a href="https://github.com/broadinstitute/gatk/blob/2ba3a15d9f49a675f38641e652418667aabc6161/src/main/java/org/broadinstitute/hellbender/utils/codecs/gtf/AbstractGtfCodec.java#L233C7-L233C60" rel="nofollow noreferrer">source</a>):</p>
  <pre><code> private static boolean validateBaseGtfFeatureFields(final GencodeGtfFeature feature) {
  
  [...]
         if (feature.getGeneType() == null) {
              return false;
          }
  [...]
  }
  </code></pre>
  <p>This function, <code>validateBaseGtfFeatureFields</code> is called as part of function <code>validateFeature</code> (<a href="https://github.com/broadinstitute/gatk/blob/2ba3a15d9f49a675f38641e652418667aabc6161/src/main/java/org/broadinstitute/hellbender/utils/codecs/gtf/AbstractGtfCodec.java#L365" rel="nofollow noreferrer">source</a>):</p>
  <pre><code>private boolean validateFeature(final GencodeGtfFeature feature) {
          return validateBaseGtfFeatureFields(feature) &amp;&amp; validateFeatureSubtype(feature);
      }
  </code></pre>
  <p>Which in turn is called when reading a GTF file and the code will exit with the error you saw if <code>validateFeatureSubtype</code> returns false (<a href="https://github.com/broadinstitute/gatk/blob/2ba3a15d9f49a675f38641e652418667aabc6161/src/main/java/org/broadinstitute/hellbender/utils/codecs/gtf/AbstractGtfCodec.java#L233C7-L233C60" rel="nofollow noreferrer">source</a>):</p>
  <pre><code>  // Now we validate our feature before returning it:
   if ( ! validateFeature(decodedFeature) ) {
      throw new UserException.MalformedFile(&quot;Decoded feature is not valid: &quot; + decodedFeature);
   }
  </code></pre>
  <p>All this means that you need to have <code>gene_type</code> set in your file. You can do this with the following <code>sed</code> command on a Linux machine:</p>
  <pre><code>sed -i.bak 's/(.*\t)/\1gene_type none; /' file.gtf 
  </code></pre>
  <p>Or, on a macOS machine, use:</p>
  <pre><code>sed -i .bak 's/(.*\t)/\1gene_type none; /' file.gtf 
  </code></pre>
  <p>This looks for the last tab (<code>\t</code>) character on each line and adds the string  <code>gene_type none</code> after it. Running it on your example file produces:</p>
  <pre><code>hla_a_01_01_01_01   IMGHLA  gene    1   3503    .   +   .   gene_type none; gene_id &quot;hla_a_01_01_01_01&quot;; gene_name &quot;hla_a_01_01_01_01&quot;; source &quot;IMGHLA&quot;;
  hla_a_01_01_01_01   IMGHLA  transcript  1   3503    .   +   .   gene_type none; gene_id &quot;hla_a_01_01_01_01&quot;; transcript_id &quot;hla_a_01_01_01_01.1&quot;; gene_name &quot;hla_a_01_01_01_01&quot;; transcript_name &quot;hla_a_01_01_01_01.1&quot;;
  hla_a_01_01_01_01   IMGHLA  exon    301 373 .   +   .   gene_type none; gene_id &quot;hla_a_01_01_01_01&quot;; transcript_id &quot;hla_a_01_01_01_01.1&quot;; gene_name &quot;hla_a_01_01_01_01&quot;; transcript_name &quot;hla_a_01_01_01_01.1&quot;; exon_number &quot;1&quot;; exon_id &quot;hla_a_01_01_01_01_e_1&quot;;
  hla_a_01_01_01_01   IMGHLA  exon    504 773 .   +   .   gene_type none; gene_id &quot;hla_a_01_01_01_01&quot;; transcript_id &quot;hla_a_01_01_01_01.1&quot;; gene_name &quot;hla_a_01_01_01_01&quot;; transcript_name &quot;hla_a_01_01_01_01.1&quot;; exon_number &quot;2&quot;; exon_id &quot;hla_a_01_01_01_01_e_2&quot;;
  hla_a_01_01_01_01   IMGHLA  exon    1015    1290    .   +   .   gene_type none; gene_id &quot;hla_a_01_01_01_01&quot;; transcript_id &quot;hla_a_01_01_01_01.1&quot;; gene_name &quot;hla_a_01_01_01_01&quot;; transcript_name &quot;hla_a_01_01_01_01.1&quot;; exon_number &quot;3&quot;; exon_id &quot;hla_a_01_01_01_01_e_3&quot;;
  </code></pre>
  <p>And this is now recognized by <code>IndexFeatureFile</code>.</p>
  <p><strong>Important</strong>: this assumes that this is the only missing element and that it is missing on <em>all</em> lines. If this is not the case, please update your question and show us a more comprehensive input example.</p>
  <p>Finally, I would consider this a bug in GATK and report it to them. The <a href="https://gmod.org/wiki/GFF3" rel="nofollow noreferrer">GTF specification</a> makes it quite clear that the entire attributes field (the last field) is optional:</p>
  <blockquote>
  <p>Column 9: “attributes”</p>
  <p>A list of feature attributes in the format tag=value. Multiple tag=value pairs are separated by semicolons. URL escaping rules are used for tags or values containing the following characters: “,=;”. Spaces are allowed in this field, but tabs must be replaced with the %09 URL escape. This field is not required.</p>
  </blockquote>
  <p>So a program failing because an optional field which is actually present simply doesn't contain one specific, also optional sub-field is very odd.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-09-18T15:59:41.313Z
Answers.LastActivityDate: 2024-09-18T15:59:41.313Z
Answers.UserDisplayName: terdon
Body: <p><em>This question was also asked on <a href="https://www.biostars.org/p/9602494/" rel="nofollow noreferrer">Biostars</a></em></p>
  <p>I created a GTF file for HLA alleles to be used as a resource for GATK Funcotator. Running Funcotator without indexing the GTF gives this error:</p>
  <pre><code>A USER ERROR has occurred: Input funcotator_dataSources.v1.7.20200521s/gencode/hla/hla.annotation.gtf must support random access to enable queries by interval. If it's a file, please index it using the bundled tool IndexFeatureFile
  </code></pre>
  <p>The first few lines of GTF file:</p>
  <pre><code>hla_a_01_01_01_01   IMGHLA  gene    1   3503    .   +   .   gene_id &quot;hla_a_01_01_01_01&quot;; gene_name &quot;hla_a_01_01_01_01&quot;; source &quot;IMGHLA&quot;;
  hla_a_01_01_01_01   IMGHLA  transcript  1   3503    .   +   .   gene_id &quot;hla_a_01_01_01_01&quot;; transcript_id &quot;hla_a_01_01_01_01.1&quot;; gene_name &quot;hla_a_01_01_01_01&quot;; transcript_name &quot;hla_a_01_01_01_01.1&quot;;
  hla_a_01_01_01_01   IMGHLA  exon    301 373 .   +   .   gene_id &quot;hla_a_01_01_01_01&quot;; transcript_id &quot;hla_a_01_01_01_01.1&quot;; gene_name &quot;hla_a_01_01_01_01&quot;; transcript_name &quot;hla_a_01_01_01_01.1&quot;; exon_number &quot;1&quot;; exon_id &quot;hla_a_01_01_01_01_e_1&quot;;
  hla_a_01_01_01_01   IMGHLA  exon    504 773 .   +   .   gene_id &quot;hla_a_01_01_01_01&quot;; transcript_id &quot;hla_a_01_01_01_01.1&quot;; gene_name &quot;hla_a_01_01_01_01&quot;; transcript_name &quot;hla_a_01_01_01_01.1&quot;; exon_number &quot;2&quot;; exon_id &quot;hla_a_01_01_01_01_e_2&quot;;
  hla_a_01_01_01_01   IMGHLA  exon    1015    1290    .   +   .   gene_id &quot;hla_a_01_01_01_01&quot;; transcript_id &quot;hla_a_01_01_01_01.1&quot;; gene_name &quot;hla_a_01_01_01_01&quot;; transcript_name &quot;hla_a_01_01_01_01.1&quot;; exon_number &quot;3&quot;; exon_id &quot;hla_a_01_01_01_01_e_3&quot;;
  </code></pre>
  <p>I need to index this file before running Funcotator. I tried using the GATK IndexFeatureFile, as suggested by Funcotator but it gives this error:</p>
  <pre><code>A USER ERROR has occurred: Unknown file is malformed: Decoded feature is not valid: hla_a_01_01_01_01   IMGHLA  gene    1   3503    .   +   .   gene_id &quot;hla_a_01_01_01_01&quot;; gene_name &quot;hla_a_01_01_01_01&quot;;  source &quot;IMGHLA&quot;;
  hla_a_01_01_01_01   IMGHLA  transcript  1   3503    .   +   .   gene_id &quot;hla_a_01_01_01_01&quot;; transcript_id &quot;hla_a_01_01_01_01.1&quot;; gene_name &quot;hla_a_01_01_01_01&quot;; transcript_name &quot;hla_a_01_01_01_01.1&quot;;
  hla_a_01_01_01_01   IMGHLA  exon    301 373 .   +   .   gene_id &quot;hla_a_01_01_01_01&quot;; transcript_id &quot;hla_a_01_01_01_01.1&quot;; gene_name &quot;hla_a_01_01_01_01&quot;; transcript_name &quot;hla_a_01_01_01_01.1&quot;; exon_number 1; exon_id &quot;hla_a_01_01_01_01_e_1&quot;;
  hla_a_01_01_01_01   IMGHLA  exon    504 773 .   +   .   gene_id &quot;hla_a_01_01_01_01&quot;; transcript_id &quot;hla_a_01_01_01_01.1&quot;; gene_name &quot;hla_a_01_01_01_01&quot;; transcript_name &quot;hla_a_01_01_01_01.1&quot;; exon_number 2; exon_id &quot;hla_a_01_01_01_01_e_2&quot;;
  hla_a_01_01_01_01   IMGHLA  exon    1015    1290    .   +   .   gene_id &quot;hla_a_01_01_01_01&quot;; transcript_id &quot;hla_a_01_01_01_01.1&quot;; gene_name &quot;hla_a_01_01_01_01&quot;; transcript_name &quot;hla_a_01_01_01_01.1&quot;; exon_number 3; exon_id &quot;hla_a_01_01_01_01_e_3&quot;;
  </code></pre>
  <p>Can someone suggest a solution for this or an alternate tool to index the GTF file?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-09-18T12:30:23.607Z
Comments.Text: I am not sure if this is the cause of your problem, but the seq_id (column1) seems to be identical to the `gene_id` value, which is unusual
Comments.UserDisplayName: Niklas
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-09-18T13:48:12.450Z
Comments.Text: Please link us to the actual file so we can check the format and also show us the full command you used. Does the file have windows line endings perhaps? Was it ever opened in Windows? Could it have spaces instead of tabs separating the fields?
Comments.UserDisplayName: terdon
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-09-18T15:13:47.760Z
Comments.Text: cross posted https://www.biostars.org/p/9602494/ gene_type is missing
Comments.UserDisplayName: Pierre
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-09-18T16:00:22.953Z
Comments.Text: Ah, thanks @Pierre! That's really weird though, why would they do that? Everything in the attributes field (including the attributes field itself) is optional so it is very off that GATK would choke on this.
Comments.UserDisplayName: terdon
Tags: <sequence-annotation><gatk><gtf><indexing>
Title: I need to index a GTF (gene transfer format) annotation file
ThreadText: # I need to index a GTF (gene transfer format) annotation file
  
  (Asked by: username_0 on 2024-09-18)
  
  *This question was also asked on [Biostars](https://www.biostars.org/p/9602494/)*
  
  I created a GTF file for HLA alleles to be used as a resource for GATK Funcotator. Running Funcotator without indexing the GTF gives this error:
  
  ```
  A USER ERROR has occurred: Input funcotator_dataSources.v1.7.20200521s/gencode/hla/hla.annotation.gtf must support random access to enable queries by interval. If it's a file, please index it using the bundled tool IndexFeatureFile
  
  ```
  
  The first few lines of GTF file:
  
  ```
  hla_a_01_01_01_01   IMGHLA  gene    1   3503    .   +   .   gene_id "hla_a_01_01_01_01"; gene_name "hla_a_01_01_01_01"; source "IMGHLA";
  hla_a_01_01_01_01   IMGHLA  transcript  1   3503    .   +   .   gene_id "hla_a_01_01_01_01"; transcript_id "hla_a_01_01_01_01.1"; gene_name "hla_a_01_01_01_01"; transcript_name "hla_a_01_01_01_01.1";
  hla_a_01_01_01_01   IMGHLA  exon    301 373 .   +   .   gene_id "hla_a_01_01_01_01"; transcript_id "hla_a_01_01_01_01.1"; gene_name "hla_a_01_01_01_01"; transcript_name "hla_a_01_01_01_01.1"; exon_number "1"; exon_id "hla_a_01_01_01_01_e_1";
  hla_a_01_01_01_01   IMGHLA  exon    504 773 .   +   .   gene_id "hla_a_01_01_01_01"; transcript_id "hla_a_01_01_01_01.1"; gene_name "hla_a_01_01_01_01"; transcript_name "hla_a_01_01_01_01.1"; exon_number "2"; exon_id "hla_a_01_01_01_01_e_2";
  hla_a_01_01_01_01   IMGHLA  exon    1015    1290    .   +   .   gene_id "hla_a_01_01_01_01"; transcript_id "hla_a_01_01_01_01.1"; gene_name "hla_a_01_01_01_01"; transcript_name "hla_a_01_01_01_01.1"; exon_number "3"; exon_id "hla_a_01_01_01_01_e_3";
  
  ```
  
  I need to index this file before running Funcotator. I tried using the GATK IndexFeatureFile, as suggested by Funcotator but it gives this error:
  
  ```
  A USER ERROR has occurred: Unknown file is malformed: Decoded feature is not valid: hla_a_01_01_01_01   IMGHLA  gene    1   3503    .   +   .   gene_id "hla_a_01_01_01_01"; gene_name "hla_a_01_01_01_01";  source "IMGHLA";
  hla_a_01_01_01_01   IMGHLA  transcript  1   3503    .   +   .   gene_id "hla_a_01_01_01_01"; transcript_id "hla_a_01_01_01_01.1"; gene_name "hla_a_01_01_01_01"; transcript_name "hla_a_01_01_01_01.1";
  hla_a_01_01_01_01   IMGHLA  exon    301 373 .   +   .   gene_id "hla_a_01_01_01_01"; transcript_id "hla_a_01_01_01_01.1"; gene_name "hla_a_01_01_01_01"; transcript_name "hla_a_01_01_01_01.1"; exon_number 1; exon_id "hla_a_01_01_01_01_e_1";
  hla_a_01_01_01_01   IMGHLA  exon    504 773 .   +   .   gene_id "hla_a_01_01_01_01"; transcript_id "hla_a_01_01_01_01.1"; gene_name "hla_a_01_01_01_01"; transcript_name "hla_a_01_01_01_01.1"; exon_number 2; exon_id "hla_a_01_01_01_01_e_2";
  hla_a_01_01_01_01   IMGHLA  exon    1015    1290    .   +   .   gene_id "hla_a_01_01_01_01"; transcript_id "hla_a_01_01_01_01.1"; gene_name "hla_a_01_01_01_01"; transcript_name "hla_a_01_01_01_01.1"; exon_number 3; exon_id "hla_a_01_01_01_01_e_3";
  
  ```
  
  Can someone suggest a solution for this or an alternate tool to index the GTF file?
  
  **Tags:** sequence-annotation, gatk, gtf, indexing
  
  > **Comments:**
  > **username_1**: I am not sure if this is the cause of your problem, but the seq_id (column1) seems to be identical to the `gene_id` value, which is unusual  
  > **username_2**: Please link us to the actual file so we can check the format and also show us the full command you used. Does the file have windows line endings perhaps? Was it ever opened in Windows? Could it have spaces instead of tabs separating the fields?  
  > **username_3**: cross posted https://www.biostars.org/p/9602494/ gene_type is missing  
  > **username_2**: Ah, thanks @username_3! That's really weird though, why would they do that? Everything in the attributes field (including the attributes field itself) is optional so it is very off that GATK would choke on this.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2024-09-18. Score: 1
  
  As username_3 Lindenbaum pointed out in [your duplicate question](https://www.biostars.org/p/9602494/) on Biostars, GATK's function that checks the validity of a GTF feature fails if `gene_type` is missing ([source](https://github.com/broadinstitute/gatk/blob/2ba3a15d9f49a675f38641e652418667aabc6161/src/main/java/org/broadinstitute/hellbender/utils/codecs/gtf/AbstractGtfCodec.java#L233C7-L233C60)):
  
  ```
   private static boolean validateBaseGtfFeatureFields(final GencodeGtfFeature feature) {
  
  [...]
         if (feature.getGeneType() == null) {
              return false;
          }
  [...]
  }
  
  ```
  
  This function, `validateBaseGtfFeatureFields` is called as part of function `validateFeature` ([source](https://github.com/broadinstitute/gatk/blob/2ba3a15d9f49a675f38641e652418667aabc6161/src/main/java/org/broadinstitute/hellbender/utils/codecs/gtf/AbstractGtfCodec.java#L365)):
  
  ```
  private boolean validateFeature(final GencodeGtfFeature feature) {
          return validateBaseGtfFeatureFields(feature) && validateFeatureSubtype(feature);
      }
  
  ```
  
  Which in turn is called when reading a GTF file and the code will exit with the error you saw if `validateFeatureSubtype` returns false ([source](https://github.com/broadinstitute/gatk/blob/2ba3a15d9f49a675f38641e652418667aabc6161/src/main/java/org/broadinstitute/hellbender/utils/codecs/gtf/AbstractGtfCodec.java#L233C7-L233C60)):
  
  ```
    // Now we validate our feature before returning it:
   if ( ! validateFeature(decodedFeature) ) {
      throw new UserException.MalformedFile("Decoded feature is not valid: " + decodedFeature);
   }
  
  ```
  
  All this means that you need to have `gene_type` set in your file. You can do this with the following `sed` command on a Linux machine:
  
  ```
  sed -i.bak 's/(.*\t)/\1gene_type none; /' file.gtf 
  
  ```
  
  Or, on a macOS machine, use:
  
  ```
  sed -i .bak 's/(.*\t)/\1gene_type none; /' file.gtf 
  
  ```
  
  This looks for the last tab (`\t`) character on each line and adds the string `gene_type none` after it. Running it on your example file produces:
  
  ```
  hla_a_01_01_01_01   IMGHLA  gene    1   3503    .   +   .   gene_type none; gene_id "hla_a_01_01_01_01"; gene_name "hla_a_01_01_01_01"; source "IMGHLA";
  hla_a_01_01_01_01   IMGHLA  transcript  1   3503    .   +   .   gene_type none; gene_id "hla_a_01_01_01_01"; transcript_id "hla_a_01_01_01_01.1"; gene_name "hla_a_01_01_01_01"; transcript_name "hla_a_01_01_01_01.1";
  hla_a_01_01_01_01   IMGHLA  exon    301 373 .   +   .   gene_type none; gene_id "hla_a_01_01_01_01"; transcript_id "hla_a_01_01_01_01.1"; gene_name "hla_a_01_01_01_01"; transcript_name "hla_a_01_01_01_01.1"; exon_number "1"; exon_id "hla_a_01_01_01_01_e_1";
  hla_a_01_01_01_01   IMGHLA  exon    504 773 .   +   .   gene_type none; gene_id "hla_a_01_01_01_01"; transcript_id "hla_a_01_01_01_01.1"; gene_name "hla_a_01_01_01_01"; transcript_name "hla_a_01_01_01_01.1"; exon_number "2"; exon_id "hla_a_01_01_01_01_e_2";
  hla_a_01_01_01_01   IMGHLA  exon    1015    1290    .   +   .   gene_type none; gene_id "hla_a_01_01_01_01"; transcript_id "hla_a_01_01_01_01.1"; gene_name "hla_a_01_01_01_01"; transcript_name "hla_a_01_01_01_01.1"; exon_number "3"; exon_id "hla_a_01_01_01_01_e_3";
  
  ```
  
  And this is now recognized by `IndexFeatureFile`.
  
  **Important**: this assumes that this is the only missing element and that it is missing on *all* lines. If this is not the case, please update your question and show us a more comprehensive input example.
  
  Finally, I would consider this a bug in GATK and report it to them. The [GTF specification](https://gmod.org/wiki/GFF3) makes it quite clear that the entire attributes field (the last field) is optional:
  
  > Column 9: “attributes”
  >
  > A list of feature attributes in the format tag=value. Multiple tag=value pairs are separated by semicolons. URL escaping rules are used for tags or values containing the following characters: “,=;”. Spaces are allowed in this field, but tabs must be replaced with the %09 URL escape. This field is not required.
  
  So a program failing because an optional field which is actually present simply doesn't contain one specific, also optional sub-field is very odd.
  
  
  ---
  
  