Answers.Body: <p>Liftover is hard. While many variants can be lifted over relatively easily, some cannot. First, some variants are not even variants in the other genome (the other genome has the variant allele as the reference). Then you can have things that were assembly errors that have been corrected, and also regions that were simply not covered in the old genome. It just isn't a straightforward problem at all.</p>
  <p>A few years ago, in 2017, I tried three separate tools, UCSC liftover, NCBI Remap, and a custom tool built by the company I worked for. I used it to lift over bed regions from hg19 to hg38. I got three separate results. Each of the three tools gave me slightly different output which means that in the best case scenario, one of them was &quot;correct&quot; and two &quot;wrong&quot;, but more likely all were &quot;wrong&quot; in different ways.</p>
  <p>The only safe way I, personally, have found is to convert variants to protein notation, lift over, and check if the new variant still results in the same protein. That cannot be done for non coding variants, of course.</p>
  <p>I found a paper, which I have only glanced through, that compares different liftover tools and suggests a combination of approaches. You might want to have a look: <a href="https://doi.org/10.1093%2Fnargab%2Flqaa054" rel="nofollow noreferrer">Luu PL, Ong PT, Dinh TP, Clark SJ. <em>Benchmark study comparing liftover tools for genome conversion of epigenome sequencing data</em>. NAR Genom Bioinform. 2020 Aug 6;2(3):lqaa054. doi: 10.1093/nargab/lqaa054</a>.</p>
  <p>Even there though, and even after limiting to only ungapped blocks, they couldn't get perfect, 100% liftover:</p>
  <blockquote>
  <p>We showed that 100.0% ungapped blocks were successfully converted (Figure ​(Figure2B),2B), of which 99.7% were lifted to the correct corresponding region on hg38.</p>
  </blockquote>
  <p>And, while they do seem to have been quite successful, they also point oiut there is no way to get this 100% perfect:</p>
  <blockquote>
  <p>Liftover is a rational solution for conversion between genome assemblies by coordinates when there is a lack of data storage, computing resource, time or human resource as it can reproduce highly accurate results for downstream analysis of epigenome sequencing data. However, it is worth noting that liftover can only assure the accuracy of coordinates and annotation conversion, therefore assumptions of achieving equivalent corresponding sequences genome-wide should not always be made. We clarify the principle underlying this powerful tool and propose a strategy to overcome the limitations of liftover at problematic regions.</p>
  </blockquote>
  <hr />
  <p><strong>WARNING</strong>: shameless plug of own software below.</p>
  <p>If you have a budget for this, you can use <a href="https://api.varsome.com/" rel="nofollow noreferrer">the API</a> made by the company I work for to convert your hg19 variants to p. notation and then query them in hg38 to do the conversion. That should correctly lift your coding variants over, but still won't really help with the non-coding ones. Plus, this is not a free tool, so it is only helpful if you are able to pay to use it.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-04-10T14:51:49.730Z
Answers.Text: Thank you @terdon for the elaborate explanation. I indeed understand the problem is hard. I have read many comparison papers and never thought any of the tools accuracy or lifting would be possible for a 100%. I was actually just hoping to be able to liftover something, rather than nothing :'-)
Answers.UserDisplayName: Dandelion
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-04-10T14:56:18.363Z
Answers.Text: @Dandelion then that paper I linked to should help a lot. There's a nice table with different tools included there which should give you something to work with. If you remember, it would be great if you could come back and post an answer with the solution you ended up choosing. I'd like to know about it too!
Answers.UserDisplayName: terdon
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-04-10T12:11:28.353Z
Answers.LastActivityDate: 2024-04-10T12:11:28.353Z
Answers.UserDisplayName: terdon
Answers.Body: <p>Could you use the hg19 --&gt; hg38 <a href="https://hgdownload.cse.ucsc.edu/goldenpath/hg19/liftOver/hg19ToHg38.over.chain.gz" rel="nofollow noreferrer">chain</a> from <a href="https://hgdownload.cse.ucsc.edu/goldenpath/hg19/liftOver/" rel="nofollow noreferrer">here</a>?</p>
  <p>As I understand it, the main difference between GrCH37 and hg19 are the format the chromosomes are written (whether there is a &quot;chr&quot; appended in front or not), and that they are mostly interchangeable, although there are of course other small differences. Unless you care about those small differences can you just use the hg19 chain file, and use a homemade script to add/subtract the &quot;chr&quot; from the input/output as needed?</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-04-11T07:16:57.067Z
Answers.Text: I have used this approach too. However, it is not simply the case of adding "chr" to the names. As you can see from the analysis from GATK (I added the link in the original post) there are differences between sequences as well. Unfortunately, for our specific case, also chromsomes we are particularly interested in.
Answers.UserDisplayName: Dandelion
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-04-10T18:48:08.050Z
Answers.LastActivityDate: 2024-04-10T18:48:08.050Z
Answers.UserDisplayName: Sarah
Body: <p>I have been tackling this problem for a while now, and I hope someone has a better answer.</p>
  <p>I need to liftover variants (.vcf) from NCBI hg37 to NCBI hg38. The NCBI remapping tool has been depreciated. For liftover you can use multiple tools, but they all need a chain file, or they only liftover annotation files like gff (like <a href="https://github.com/wurmlab/flo" rel="nofollow noreferrer">flo</a>). The only chain files available (that I can find) are from UCSC and ENSEMBL, and as far as I know, these reference genomes <a href="https://gatk.broadinstitute.org/hc/en-us/articles/360035890711-GRCh37-hg19-b37-humanG1Kv37-Human-Reference-Discrepancies" rel="nofollow noreferrer">are different</a>.</p>
  <p>I have found some very oblique ways to make chain files <a href="https://hgwdev.gi.ucsc.edu/%7Ekent/src/unzipped/hg/doc/liftOver.txt" rel="nofollow noreferrer">here</a>, <a href="http://genomewiki.ucsc.edu/index.php?title=Minimal_Steps_For_LiftOver" rel="nofollow noreferrer">here</a>, and <a href="https://www.biostars.org/p/391080/#465890" rel="nofollow noreferrer">here</a>. When I finally figured out what was going on, the steps seemed straightforward, but when I try to follow them the amount of additional steps you need to take becomes exponential almost.</p>
  <p>At a certain point I also found some perl script from NCBI for remapping or getting the files I need (sorry, can't find the link), which I of course didn't get working because every step I took another dependency failed.</p>
  <p>I also tried contacting NCBI, to no avail.</p>
  <p>Lifting over is essentially aligning one reference genome to another. Until here I conceptually get it.</p>
  <p>I cannot believe this is so hard and I am tired :'-). There must be someone who has already figured this out and knows the easier way to do this. My research should not be about &quot;the way to lift over an NCBI genome&quot;. Before I continue this struggle (mainly trying to finish making a chain file as described in the links above), does anyone have anything useful for me? Am I missing something?</p>
  <p>Thank you in advance</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-04-10T14:57:09.750Z
Comments.Text: I don't think ENSEMBL have their own genome version, are you sure?
Comments.UserDisplayName: terdon
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-04-11T07:12:06.433Z
Comments.Text: There seem to be differences between the genome provided by ensembl. It is explained more in the link I provided in my post.
Comments.UserDisplayName: Dandelion
Tags: <vcf><ncbi><liftover>
Title: Liftover variants/vcf called on NCBI reference genome
ThreadText: # Liftover variants/vcf called on NCBI reference genome
  
  (Asked by: username_0 on 2024-04-10)
  
  I have been tackling this problem for a while now, and I hope someone has a better answer.
  
  I need to liftover variants (.vcf) from NCBI hg37 to NCBI hg38. The NCBI remapping tool has been depreciated. For liftover you can use multiple tools, but they all need a chain file, or they only liftover annotation files like gff (like [flo](https://github.com/wurmlab/flo)). The only chain files available (that I can find) are from UCSC and ENSEMBL, and as far as I know, these reference genomes [are different](https://gatk.broadinstitute.org/hc/en-us/articles/360035890711-GRCh37-hg19-b37-humanG1Kv37-Human-Reference-Discrepancies).
  
  I have found some very oblique ways to make chain files [here](https://hgwdev.gi.ucsc.edu/%7Ekent/src/unzipped/hg/doc/liftOver.txt), [here](http://genomewiki.ucsc.edu/index.php?title=Minimal_Steps_For_LiftOver), and [here](https://www.biostars.org/p/391080/#465890). When I finally figured out what was going on, the steps seemed straightforward, but when I try to follow them the amount of additional steps you need to take becomes exponential almost.
  
  At a certain point I also found some perl script from NCBI for remapping or getting the files I need (sorry, can't find the link), which I of course didn't get working because every step I took another dependency failed.
  
  I also tried contacting NCBI, to no avail.
  
  Lifting over is essentially aligning one reference genome to another. Until here I conceptually get it.
  
  I cannot believe this is so hard and I am tired :'-). There must be someone who has already figured this out and knows the easier way to do this. My research should not be about "the way to lift over an NCBI genome". Before I continue this struggle (mainly trying to finish making a chain file as described in the links above), does anyone have anything useful for me? Am I missing something?
  
  Thank you in advance
  
  **Tags:** vcf, ncbi, liftover
  
  > **Comments:**
  > **username_1**: I don't think ENSEMBL have their own genome version, are you sure?  
  > **username_0**: There seem to be differences between the genome provided by ensembl. It is explained more in the link I provided in my post.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2024-04-10. Score: 2
  
  Liftover is hard. While many variants can be lifted over relatively easily, some cannot. First, some variants are not even variants in the other genome (the other genome has the variant allele as the reference). Then you can have things that were assembly errors that have been corrected, and also regions that were simply not covered in the old genome. It just isn't a straightforward problem at all.
  
  A few years ago, in 2017, I tried three separate tools, UCSC liftover, NCBI Remap, and a custom tool built by the company I worked for. I used it to lift over bed regions from hg19 to hg38. I got three separate results. Each of the three tools gave me slightly different output which means that in the best case scenario, one of them was "correct" and two "wrong", but more likely all were "wrong" in different ways.
  
  The only safe way I, personally, have found is to convert variants to protein notation, lift over, and check if the new variant still results in the same protein. That cannot be done for non coding variants, of course.
  
  I found a paper, which I have only glanced through, that compares different liftover tools and suggests a combination of approaches. You might want to have a look: [Luu PL, Ong PT, Dinh TP, Clark SJ. *Benchmark study comparing liftover tools for genome conversion of epigenome sequencing data*. NAR Genom Bioinform. 2020 Aug 6;2(3):lqaa054. doi: 10.1093/nargab/lqaa054](https://doi.org/10.1093%2Fnargab%2Flqaa054).
  
  Even there though, and even after limiting to only ungapped blocks, they couldn't get perfect, 100% liftover:
  
  > We showed that 100.0% ungapped blocks were successfully converted (Figure ​(Figure2B),2B), of which 99.7% were lifted to the correct corresponding region on hg38.
  
  And, while they do seem to have been quite successful, they also point oiut there is no way to get this 100% perfect:
  
  > Liftover is a rational solution for conversion between genome assemblies by coordinates when there is a lack of data storage, computing resource, time or human resource as it can reproduce highly accurate results for downstream analysis of epigenome sequencing data. However, it is worth noting that liftover can only assure the accuracy of coordinates and annotation conversion, therefore assumptions of achieving equivalent corresponding sequences genome-wide should not always be made. We clarify the principle underlying this powerful tool and propose a strategy to overcome the limitations of liftover at problematic regions.
  
  ---
  
  **WARNING**: shameless plug of own software below.
  
  If you have a budget for this, you can use [the API](https://api.varsome.com/) made by the company I work for to convert your hg19 variants to p. notation and then query them in hg38 to do the conversion. That should correctly lift your coding variants over, but still won't really help with the non-coding ones. Plus, this is not a free tool, so it is only helpful if you are able to pay to use it.
  
  > **Comments:**
  > **username_0**: Thank you @username_1 for the elaborate explanation. I indeed understand the problem is hard. I have read many comparison papers and never thought any of the tools accuracy or lifting would be possible for a 100%. I was actually just hoping to be able to liftover something, rather than nothing :'-)  
  > **username_1**: @username_0 then that paper I linked to should help a lot. There's a nice table with different tools included there which should give you something to work with. If you remember, it would be great if you could come back and post an answer with the solution you ended up choosing. I'd like to know about it too!  
  
  ---
  
  ### Answer by username_2 on 2024-04-10. Score: 1
  
  Could you use the hg19 --> hg38 [chain](https://hgdownload.cse.ucsc.edu/goldenpath/hg19/liftOver/hg19ToHg38.over.chain.gz) from [here](https://hgdownload.cse.ucsc.edu/goldenpath/hg19/liftOver/)?
  
  As I understand it, the main difference between GrCH37 and hg19 are the format the chromosomes are written (whether there is a "chr" appended in front or not), and that they are mostly interchangeable, although there are of course other small differences. Unless you care about those small differences can you just use the hg19 chain file, and use a homemade script to add/subtract the "chr" from the input/output as needed?
  
  > **Comments:**
  > **username_0**: I have used this approach too. However, it is not simply the case of adding "chr" to the names. As you can see from the analysis from GATK (I added the link in the original post) there are differences between sequences as well. Unfortunately, for our specific case, also chromsomes we are particularly interested in.  
  
  ---
  
  