Answers.Body: <p>There is one very simplistic way I use which <em>might</em> work for what you are doing, it is similar to what terdon proposed.</p>
  
  <p>Take a de-novo microbial genome annotation tool (I have my own, but you could use/modify <a href="http://www.vicbioinformatics.com/software.prokka.shtml" rel="nofollow noreferrer">prokka</a>). Tools like these often first predict gene boundaries (with other tools like <a href="http://prodigal.ornl.gov/" rel="nofollow noreferrer">prodigal</a> or <a href="https://ccb.jhu.edu/software/glimmer/" rel="nofollow noreferrer">glimmer</a>) and then try to assign a function to found genes. This function assignment is often done with BLAST and other tools ... and that is where you can go in and modify to do what you need.</p>
  
  <p>I use a "knowledge" protein database of genes I want to have very strictly annotated as a first line of annotation (e.g. in your case: the annotated genomes). For that I loop through very strict identity/similarity parameters which get gradually relaxed.</p>
  
  <p>E.g.:
  Loop 0: only transfer annotations at 100% DNA identity, same length.
  Loop 1: only transfer annotations at 100% similarity, same length.
  Loop 2: only transfer annotations at 99% similarity, length +/- 1%.
  ...
  Loop n: only transfer annotations at 100-(n-1)% similarity, length +/- (n-1)%.</p>
  
  <p>In each loop, obviously only annotate what has not been annotated in previous loops.</p>
  
  <p>After that, use "normal" annotation pipeline of the tool to annotate the rest.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-05-31T16:33:32.340Z
Answers.Text: Doesn't that require the target genome's genes to have been found first? Or  can your tool also do de-novo gene prediction? (sounds like a very useful tool, by the way, kudos!)
Answers.UserDisplayName: terdon
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-05-31T17:27:55.340Z
Answers.Text: Prokaryotik gene finding/prediction is a more or less solved problem, existing tools work reasonably well. See http://prodigal.ornl.gov/ and http://prodigal.ornl.gov/ (just to name two).
Answers.UserDisplayName: BaCh
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-05-31T17:31:36.500Z
Answers.Text: Yes, I know, I was just surprised you didn't mention that in your answer. If I understand correctly, the first step would be for the OP to find the list of putative genes in their newly sequenced genome, right?
Answers.UserDisplayName: terdon
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-05-31T17:37:28.263Z
Answers.Text: Correct. Prokka (http://www.vicbioinformatics.com/software.prokka.shtml) uses a whole battery of third-party tools (including prodigal) to annotate a genome de-novo, I started by modifying prokka before I wrote my own one, which uses some ideas from the prokka pipeline.
Answers.UserDisplayName: BaCh
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-05-31T15:34:47.833Z
Answers.LastActivityDate: 2017-05-31T17:43:32.240Z
Answers.LastEditDate: 2017-05-31T17:43:32.240Z
Answers.UserDisplayName: BaCh
Answers.Body: <p>I think you will have to first identify the regions homologous to the ones defined in your GFF and then transfer the annotations. Of course, the assumption there is that the homolog will also have the same annotation which is often not true. However, I don't see how you can do it in any other way since you cannot use genomic coordinates (and you would still be making the same assumption even if you could, anyway) when the genomes are so different.</p>
  <p>For a very simplistic approach (which might be enough if, as you say, your sequences are almost identical), you can do something like:</p>
  <ol>
  <li><p>Collect the sequences of interest from your already annotated species.</p>
  </li>
  <li><p>Use a tool like <a href="https://www.ebi.ac.uk/Tools/psa/genewise/" rel="nofollow noreferrer"><code>genewise</code></a> or <a href="https://www.ebi.ac.uk/about/vertebrate-genomics/software/exonerate" rel="nofollow noreferrer"><code>exonerate</code></a> to map these into the target genome. Both tools can return gff-formatted output and both can find multiple hits in the target genome. For what you want, I would suggest using a very high threshold of sequence similarity and query coverage (where the target sequence found covers all or most of the query sequence used).</p>
  <p>Since these are microbial genomes and therefore splicing isn't a problem, you could do the same thing with even a simple BLASTn or tBLASTn if you start from protein sequences.</p>
  </li>
  <li><p>At this point, you should have a list of homologs (some of which will be <a href="https://biology.stackexchange.com/a/4964/1306">orthologs and others paralogs</a>) and you can transfer the annotations of the query sequence over to the target.</p>
  </li>
  </ol>
  <p>Again, I stress that this is making a whopping huge assumption: that homologous sequences have the same function and can automatically be annotated as whatever you had in the query genome. This is going to be true for many cases but it will also be false for others. Especially if you are looking at paralogs (genes whose duplication occurred after the speciation event and are therefore likely to have diverged in function).</p>
  <p>However, as I said before, this problem would be exactly the same even if you did manage to transfer annotations just by identifying the syntenic regions of the genomes<sup>1</sup>, so there's not much difference there.</p>
  <hr />
  <p><sup>1</sup> <sub>As I said in the comments, I don't see how this could be possible. By definition, if you have extensive duplications, the genomic coordinates will be completely different and it is impossible to map from one genome into the other. </sub></p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-05-31T14:04:22.683Z
Answers.LastActivityDate: 2017-05-31T14:04:22.683Z
Answers.LastEditDate: 2020-06-18T08:30:41.743Z
Answers.UserDisplayName: terdon
Body: <p>Microbial genomes can contain extensive duplications. Often we'd like to transfer annotations from an annotated species to one that is newly sequenced. </p>
  
  <p>Existing tools (e.g. RATT, LiftOver, Kraken) either make specific assumptions about how closely related the species are or fail to transfer when multiple matches are found in the new genome, especially if the sequences are highly similar. </p>
  
  <p>Specifically, I have a synthetic biology application where genes can duplicate extensively. They are identical in sequence but duplicated many times and be relocated (i.e., not just adjacent to each other). None of the above mentioned tools are able to transfer coordinates of annotations to genomes with multiple copies of features.</p>
  
  <p>Are there any pre-existing tools or software that transfer annotations in this scenario? Ideas for ways to do this robustly?</p>
  
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-05-31T11:19:55.397Z
Comments.Text: Can you provide more detail? What's your input (e.g. raw reads, assemblies, open-reading frames)? what kind of sequencing? what is the exact output you want? ' I'm not sure I understand your point about the assumptions. Also, LiftOver and Kraken are completely different tools with different uses
Comments.UserDisplayName: Chris_Rands
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-05-31T11:22:33.603Z
Comments.Text: Would this even be theoretically possible? How can you assume the annotations are transferable if there are extensive duplications? It's probably better to look for homologs instead.
Comments.UserDisplayName: terdon
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-05-31T11:25:06.843Z
Comments.Text: @terdon do you mean orthologs? homologs = orthologs (non-duplicated) + paralogs (duplicated)
Comments.UserDisplayName: Chris_Rands
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-05-31T11:54:51.840Z
Comments.Text: @Chris_Rands No, I meant homologs. Precisely because we can't know whether they are ortho- or para- (I have a nice post on the difference between the two [here](https://biology.stackexchange.com/a/4964/1306), by the way), so all you can do first is find homologs and then try and figure out whether they are similar enough to carry any annotations over.
Comments.UserDisplayName: terdon
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-05-31T12:03:19.247Z
Comments.Text: @terdon I see well resolving orthologs/paralogs is not easy of course, but it can be done, depending on the exact data (I don't know what the OP's data looks like), for example some of my colleagues maintain orthodb http://www.orthodb.org/
Comments.UserDisplayName: Chris_Rands
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-05-31T13:35:49.587Z
Comments.Text: Oh, of course it can be done! My point was that looking for regions of homology (of whatever type) seems like a better way of transferring annotations than attempting to translate genomic coordinates between genomes of different species.
Comments.UserDisplayName: terdon
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-05-31T13:43:04.360Z
Comments.Text: @Chris_Rands: Input would be assemblies, e.g. de novo from gDNA sequencing. Output would be a transfer of annotations (e.g. gff format) from characterized species to newly assembled genome (coordinate transfer). Both LiftOver and Kraken (this one, just to make sure were on same page: https://github.com/nedaz/kraken) do this. LiftOver more appropriate for coordinate transfer between closely related sequences e.g. different assemblies; Kraken uses genome alignment (MUMer, Satsuma) so better for more divergent sequences.
Comments.UserDisplayName: scalefreegan
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-05-31T13:49:24.283Z
Comments.Text: @terdon: distinguishing between type/origin of homology would go beyond scope of what I would want to accomplish, but the difference is important as you point out. also right to say that transferring smaller homologous regions would be better, especially for diverged species. fyi about application: I have a syn bio application where genes can duplicate extensively. they are identical in sequence but duplicated many times and relocated (i.e. not adjacent). None of the above mentioned tools was able to transfer coordinates of annotations to genomes with multiple copies of annotation.
Comments.UserDisplayName: scalefreegan
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-05-31T13:54:42.937Z
Comments.Text: Yes, nor would I expect them to. That's what I was saying. The liftover tools simply map coordinates, they won't be able to deal with this sort of thing. I am afraid you will have to do it manually by getting a list of genes/proteins of interest, finding their homologs and transferring the annotations over (with the obvious caveats about whether or not you can assume the annotations are transferable). Won't be much fun, unfortunately.
Comments.UserDisplayName: terdon
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-05-31T13:55:45.607Z
Comments.Text: apologies I thought you meant kraken: http://ccb.jhu.edu/software/kraken/, who names these tools? anyway, this is quite non-trivial to do properly. you'll need to do genome assembly, gene predictions and ortholog/paralog assignment; there are various pipelines (some reviewed here: https://www.ncbi.nlm.nih.gov/pubmed/27043882), but they'll take some time. alternatively, for something more 'quick and dirty', @terdon 's suggestions seems sensible
Comments.UserDisplayName: Chris_Rands
Tags: <sequence-annotation><ngs>
Title: How to transfer gff annotations in genome with extensive duplications?
ThreadText: # How to transfer gff annotations in genome with extensive duplications?
  
  (Asked by: username_0 on 2017-05-31)
  
  Microbial genomes can contain extensive duplications. Often we'd like to transfer annotations from an annotated species to one that is newly sequenced.
  
  Existing tools (e.g. RATT, LiftOver, Kraken) either make specific assumptions about how closely related the species are or fail to transfer when multiple matches are found in the new genome, especially if the sequences are highly similar.
  
  Specifically, I have a synthetic biology application where genes can duplicate extensively. They are identical in sequence but duplicated many times and be relocated (i.e., not just adjacent to each other). None of the above mentioned tools are able to transfer coordinates of annotations to genomes with multiple copies of features.
  
  Are there any pre-existing tools or software that transfer annotations in this scenario? Ideas for ways to do this robustly?
  
  **Tags:** sequence-annotation, ngs
  
  > **Comments:**
  > **username_1**: Can you provide more detail? What's your input (e.g. raw reads, assemblies, open-reading frames)? what kind of sequencing? what is the exact output you want? ' I'm not sure I understand your point about the assumptions. Also, LiftOver and Kraken are completely different tools with different uses  
  > **username_2**: Would this even be theoretically possible? How can you assume the annotations are transferable if there are extensive duplications? It's probably better to look for homologs instead.  
  > **username_1**: @username_2 do you mean orthologs? homologs = orthologs (non-duplicated) + paralogs (duplicated)  
  > **username_2**: @username_1 No, I meant homologs. Precisely because we can't know whether they are ortho- or para- (I have a nice post on the difference between the two [here](https://biology.stackexchange.com/a/4964/1306), by the way), so all you can do first is find homologs and then try and figure out whether they are similar enough to carry any annotations over.  
  > **username_1**: @username_2 I see well resolving orthologs/paralogs is not easy of course, but it can be done, depending on the exact data (I don't know what the OP's data looks like), for example some of my colleagues maintain orthodb http://www.orthodb.org/  
  > **username_2**: Oh, of course it can be done! My point was that looking for regions of homology (of whatever type) seems like a better way of transferring annotations than attempting to translate genomic coordinates between genomes of different species.  
  > **username_0**: @username_1: Input would be assemblies, e.g. de novo from gDNA sequencing. Output would be a transfer of annotations (e.g. gff format) from characterized species to newly assembled genome (coordinate transfer). Both LiftOver and Kraken (this one, just to make sure were on same page: https://github.com/nedaz/kraken) do this. LiftOver more appropriate for coordinate transfer between closely related sequences e.g. different assemblies; Kraken uses genome alignment (MUMer, Satsuma) so better for more divergent sequences.  
  > **username_0**: @username_2: distinguishing between type/origin of homology would go beyond scope of what I would want to accomplish, but the difference is important as you point out. also right to say that transferring smaller homologous regions would be better, especially for diverged species. fyi about application: I have a syn bio application where genes can duplicate extensively. they are identical in sequence but duplicated many times and relocated (i.e. not adjacent). None of the above mentioned tools was able to transfer coordinates of annotations to genomes with multiple copies of annotation.  
  > **username_2**: Yes, nor would I expect them to. That's what I was saying. The liftover tools simply map coordinates, they won't be able to deal with this sort of thing. I am afraid you will have to do it manually by getting a list of genes/proteins of interest, finding their homologs and transferring the annotations over (with the obvious caveats about whether or not you can assume the annotations are transferable). Won't be much fun, unfortunately.  
  > **username_1**: apologies I thought you meant kraken: http://ccb.jhu.edu/software/kraken/, who names these tools? anyway, this is quite non-trivial to do properly. you'll need to do genome assembly, gene predictions and ortholog/paralog assignment; there are various pipelines (some reviewed here: https://www.ncbi.nlm.nih.gov/pubmed/27043882), but they'll take some time. alternatively, for something more 'quick and dirty', @username_2 's suggestions seems sensible  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_3 on 2017-05-31. Score: 5
  
  There is one very simplistic way I use which *might* work for what you are doing, it is similar to what username_2 proposed.
  
  Take a de-novo microbial genome annotation tool (I have my own, but you could use/modify [prokka](http://www.vicbioinformatics.com/software.prokka.shtml)). Tools like these often first predict gene boundaries (with other tools like [prodigal](http://prodigal.ornl.gov/) or [glimmer](https://ccb.jhu.edu/software/glimmer/)) and then try to assign a function to found genes. This function assignment is often done with BLAST and other tools ... and that is where you can go in and modify to do what you need.
  
  I use a "knowledge" protein database of genes I want to have very strictly annotated as a first line of annotation (e.g. in your case: the annotated genomes). For that I loop through very strict identity/similarity parameters which get gradually relaxed.
  
  E.g.:
  Loop 0: only transfer annotations at 100% DNA identity, same length.
  Loop 1: only transfer annotations at 100% similarity, same length.
  Loop 2: only transfer annotations at 99% similarity, length +/- 1%.
  ...
  Loop n: only transfer annotations at 100-(n-1)% similarity, length +/- (n-1)%.
  
  In each loop, obviously only annotate what has not been annotated in previous loops.
  
  After that, use "normal" annotation pipeline of the tool to annotate the rest.
  
  > **Comments:**
  > **username_2**: Doesn't that require the target genome's genes to have been found first? Or  can your tool also do de-novo gene prediction? (sounds like a very useful tool, by the way, kudos!)  
  > **username_3**: Prokaryotik gene finding/prediction is a more or less solved problem, existing tools work reasonably well. See http://prodigal.ornl.gov/ and http://prodigal.ornl.gov/ (just to name two).  
  > **username_2**: Yes, I know, I was just surprised you didn't mention that in your answer. If I understand correctly, the first step would be for the OP to find the list of putative genes in their newly sequenced genome, right?  
  > **username_3**: Correct. Prokka (http://www.vicbioinformatics.com/software.prokka.shtml) uses a whole battery of third-party tools (including prodigal) to annotate a genome de-novo, I started by modifying prokka before I wrote my own one, which uses some ideas from the prokka pipeline.  
  
  ---
  
  ### Answer by username_2 on 2017-05-31. Score: 3
  
  I think you will have to first identify the regions homologous to the ones defined in your GFF and then transfer the annotations. Of course, the assumption there is that the homolog will also have the same annotation which is often not true. However, I don't see how you can do it in any other way since you cannot use genomic coordinates (and you would still be making the same assumption even if you could, anyway) when the genomes are so different.
  
  For a very simplistic approach (which might be enough if, as you say, your sequences are almost identical), you can do something like:
  
  1. Collect the sequences of interest from your already annotated species.
  2. Use a tool like [`genewise`](https://www.ebi.ac.uk/Tools/psa/genewise/) or [`exonerate`](https://www.ebi.ac.uk/about/vertebrate-genomics/software/exonerate) to map these into the target genome. Both tools can return gff-formatted output and both can find multiple hits in the target genome. For what you want, I would suggest using a very high threshold of sequence similarity and query coverage (where the target sequence found covers all or most of the query sequence used).
  
     Since these are microbial genomes and therefore splicing isn't a problem, you could do the same thing with even a simple BLASTn or tBLASTn if you start from protein sequences.
  3. At this point, you should have a list of homologs (some of which will be [orthologs and others paralogs](https://biology.stackexchange.com/a/4964/1306)) and you can transfer the annotations of the query sequence over to the target.
  
  Again, I stress that this is making a whopping huge assumption: that homologous sequences have the same function and can automatically be annotated as whatever you had in the query genome. This is going to be true for many cases but it will also be false for others. Especially if you are looking at paralogs (genes whose duplication occurred after the speciation event and are therefore likely to have diverged in function).
  
  However, as I said before, this problem would be exactly the same even if you did manage to transfer annotations just by identifying the syntenic regions of the genomes1, so there's not much difference there.
  
  ---
  
  1 As I said in the comments, I don't see how this could be possible. By definition, if you have extensive duplications, the genomic coordinates will be completely different and it is impossible to map from one genome into the other.
  
  
  ---
  
  