Answers.Body: <p>The ID history converter is available as a <a href="https://github.com/Ensembl/ensembl-tools/tree/release/103/scripts/id_history_converter" rel="nofollow noreferrer">Perl script</a>.</p>
  <p>This script accesses these <a href="http://www.ensembl.org/info/docs/api/core/core_schema.html#gene_archive" rel="nofollow noreferrer">MySQL tables</a>, so you could query these tables directly.</p>
  <p>Note that the mapping session is only one release at a time, so you would need to run 25 queries to go from release 75 to 100.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-22T14:38:04.890Z
Answers.Text: Thank you for your answer. To use the perl script, I wonder if there is any tutorial to refer to. I am particularly interested in knowing how to "tell" the script to convert from a given specific release to other specific release e.g. from 75 to 76.
Answers.UserDisplayName: user345394
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-22T14:57:50.050Z
Answers.Text: It just gives you the whole history, you would need to parse out the releases you want.
Answers.UserDisplayName: Emily_Ensembl
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-25T20:17:13.593Z
Answers.Text: `IDmapper.pl` does the job, but I wish there was an easy way to install its dependency called `Bio::EnsEMBL::Registry`. I feel that perhaps a wrapper of sorts, for example in the Anaconda system, would be immensely helpful. (Please somebody make it happen! :D )
Answers.UserDisplayName: user345394
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-22T13:51:44.683Z
Answers.LastActivityDate: 2021-03-22T13:58:59.217Z
Answers.LastEditDate: 2021-03-22T13:58:59.217Z
Answers.UserDisplayName: Emily_Ensembl
Answers.Body: <p>Here's one way to do it programmatically using the biomaRt package (your potential solution #1):</p>
  <pre><code>library(biomaRt)
  library(dplyr)
  
  mart37 &lt;- useMart(biomart = &quot;ENSEMBL_MART_ENSEMBL&quot;,
                    host    = &quot;grch37.ensembl.org&quot;,
                    path    = &quot;/biomart/martservice&quot;,
                    dataset = &quot;hsapiens_gene_ensembl&quot;)
  
  mart38 &lt;- useMart(biomart = &quot;ENSEMBL_MART_ENSEMBL&quot;,
                    path    = &quot;/biomart/martservice&quot;,
                    dataset = &quot;hsapiens_gene_ensembl&quot;)
  
  #pick all genes on chr22 from build 37 as a test dataset::
  out37 &lt;- getBM(attributes = c('hgnc_symbol', 'chromosome_name',
                                'ensembl_gene_id', 'ensembl_transcript_id', 'ccds'),
                    filters = c('chromosome_name'),
                    values = list(&quot;22&quot;),
                    mart = mart37) %&gt;% #empty string CCDS ID's
    filter(ccds != &quot;&quot;)
  
  #NOTE:: to check which filters can be used to query mart37, run this command:
  #View(listFilters(mart37))
  
  out38 &lt;- getBM(attributes = c('hgnc_symbol', 'chromosome_name',
                                'ensembl_gene_id', 'ensembl_transcript_id', 'ccds'),
                          filters = c('ccds'),
                          values = list(out37$ccds),
                          mart = mart38)
  </code></pre>
  <p>What this does is query the build 38 DB via the CCDS ID's of build 37. You can include other variables in this <code>out38</code> dataframe by simply adding to the attributes argument above. And in order to see all of the potential variables that can be included, you just have to run <code>View(listFilters(mart37))</code> or <code>View(listFilters(mart38))</code>, depending on which build you are interested in.</p>
  <p>Does this solve your problem?</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-25T22:04:02.017Z
Answers.Text: your solution could indeed be useful. I am just curious which ensembl release numbers are used when `mart37` and `mart38` were built. Is there a way to specify that?
Answers.UserDisplayName: user345394
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-26T13:10:34.870Z
Answers.Text: `mart37` corresponds to the GRCh37 release, while `mart38` corresponds to GRCh38. This is specified in the `host=` argument in the code above. `mart38` is the default, and so is not specified. Does this answer your question?
Answers.UserDisplayName: h3ab74
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-26T13:56:09.347Z
Answers.Text: P.S. if you're referring to the version number of the ensembl release, you can obtain this by adding `'ensembl_gene_id_version'` to the `attributes` array in the code above.
Answers.UserDisplayName: h3ab74
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-26T22:53:22.967Z
Answers.Text: By Ensembl release number I meant [version numbers of the assemblies maintained by ensembl](https://www.ensembl.info/category/01-release/), e.g. release 75 corresponds to GRCh37 for human genome.
Answers.UserDisplayName: user345394
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-26T22:57:28.020Z
Answers.Text: `ensembl_gene_id_version` could be useful if it is possible to know to which release that belongs to. e.g. how would I know if `x.3` id belongs to genome release 90 or 91? To get this information, I would have to run 'Ensembl IDmapper' which I would like to avoid, because of the issue I mentioned in my question.
Answers.UserDisplayName: user345394
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-25T16:15:02.477Z
Answers.LastActivityDate: 2021-03-25T16:15:02.477Z
Answers.UserDisplayName: h3ab74
Body: <p>3 solutions that I had to unfortunately rule out.</p>
  <p><strong>Potential solution #1</strong></p>
  <p>As far as I know, the conventional way is to do this is by using CCDS ids (e.g. using biomart). However, the issue is that, as I see on the <a href="https://www.ncbi.nlm.nih.gov/projects/CCDS/CcdsBrowse.cgi?REQUEST=SHOW_STATISTICS" rel="nofollow noreferrer">CCDS project page</a>, the last release of CCDS ids was made almost 3 years ago (!) and it seems to be compatible with Ensembl release 92. Then how should one convert ids to the latest Ensembl release e.g. 100?</p>
  <div class="s-table-container">
  <table class="s-table">
  <thead>
  <tr>
  <th>CCDS Release</th>
  <th>Species</th>
  <th>NCBI Annotation Release</th>
  <th>Ensembl Annotation Release</th>
  <th>Assembly Name</th>
  <th>Assembly ID</th>
  <th>Made Public</th>
  </tr>
  </thead>
  <tbody>
  <tr>
  <td>22</td>
  <td>Homo sapiens</td>
  <td>109</td>
  <td>92</td>
  <td>GRCh38.p12</td>
  <td>GCF_000001405.38</td>
  <td>06/14/2018</td>
  </tr>
  </tbody>
  </table>
  </div>
  <p><strong>Potential solution #2</strong></p>
  <p>I am also aware of Ensembl's <a href="http://useast.ensembl.org/Homo_sapiens/Tools/IDMapper" rel="nofollow noreferrer">ID History Converter</a>. However, I am looking for a programmatic approach. Also, I noticed that if I try to convert ~&gt;20K ids with ID History Converter, in my experience, there is always some issue. e.g. an error I got when I tried to convert ~70K ensembl transcript ids.
  <a href="https://i.sstatic.net/9Ie1t.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/9Ie1t.png" alt="enter image description here" /></a></p>
  <p>As a side note, I wonder why this ID conversion functionality is not available via Ensembl's REST API. To me, id conversion seems to be a basic enough task that almost every bioinformatician has to do. So it would be immensely helpful to be someday able to do this programmatically using Ensembl's mighty a REST API.</p>
  <p><strong>Potential solution #3</strong></p>
  <p>Finally, as another potential solution, over at Ensembl's FTP website, located within the directory of the release to which I am interested in converting my ids to (release 100, target release), I saw that according to a <a href="http://ftp.ensembl.org/pub/release-100/README" rel="nofollow noreferrer">README file</a>, &quot;ensembl_archive_&quot; directory should contain &quot;Data on historical Ensembl releases&quot;. So, I wondered if this directory contains a map between ids from different releases. However, without any extra info about column names, contents, file formats etc, I couldn't really figure out how to use those files. So, I had to rule this one out too.</p>
  <pre><code>Index of /pub/release-100/mysql/ensembl_archive_100/
  ../
  CHECKSUMS                                          26-Mar-2020 14:31                 132
  ens_release.txt.gz                                 26-Mar-2020 14:31                1098
  ensembl_archive_100.sql.gz                         26-Mar-2020 14:31                 943
  release_species.txt.gz                             26-Mar-2020 14:31               39620
  species.txt.gz                                     26-Mar-2020 14:31                6669
  </code></pre>
  <p>Of note, I saw a similar &quot;mysql&quot; directory over at the directory of the release whose ids I want to convert (source release): <code>http://ftp.ensembl.org/pub/grch37/release-100/mysql/homo_sapiens_core_100_37/ </code>. However, again, I do not understand its contents without any extra information.</p>
  <p>So, finally, I am not sure what is the best way to programmatically convert Ensembl ids from between different releases. As a test case scenario, I want to convert all the protein_coding transcript ids from GRCh37 (release 75) to GRCh38 (release 100). If anybody has any suggestions about this, please kindly let me know.</p>
  
Tags: <gene><ensembl><identifiers><conversion>
Title: What is the best way to programmatically convert Ensembl ids from one release to other? e.g. transcript ids from release 75 to 100
ThreadText: # What is the best way to programmatically convert Ensembl ids from one release to other? e.g. transcript ids from release 75 to 100
  
  (Asked by: username_0 on 2021-03-19)
  
  3 solutions that I had to unfortunately rule out.
  
  **Potential solution #1**
  
  As far as I know, the conventional way is to do this is by using CCDS ids (e.g. using biomart). However, the issue is that, as I see on the [CCDS project page](https://www.ncbi.nlm.nih.gov/projects/CCDS/CcdsBrowse.cgi?REQUEST=SHOW_STATISTICS), the last release of CCDS ids was made almost 3 years ago (!) and it seems to be compatible with Ensembl release 92. Then how should one convert ids to the latest Ensembl release e.g. 100?
  
  | CCDS Release | Species | NCBI Annotation Release | Ensembl Annotation Release | Assembly Name | Assembly ID | Made Public |
  | --- | --- | --- | --- | --- | --- | --- |
  | 22 | Homo sapiens | 109 | 92 | GRCh38.p12 | GCF\_000001405.38 | 06/14/2018 |
  
  **Potential solution #2**
  
  I am also aware of Ensembl's [ID History Converter](http://useast.ensembl.org/Homo_sapiens/Tools/IDMapper). However, I am looking for a programmatic approach. Also, I noticed that if I try to convert ~>20K ids with ID History Converter, in my experience, there is always some issue. e.g. an error I got when I tried to convert ~70K ensembl transcript ids.
  [![enter image description here](https://i.sstatic.net/9Ie1t.png)](https://i.sstatic.net/9Ie1t.png)
  
  As a side note, I wonder why this ID conversion functionality is not available via Ensembl's REST API. To me, id conversion seems to be a basic enough task that almost every bioinformatician has to do. So it would be immensely helpful to be someday able to do this programmatically using Ensembl's mighty a REST API.
  
  **Potential solution #3**
  
  Finally, as another potential solution, over at Ensembl's FTP website, located within the directory of the release to which I am interested in converting my ids to (release 100, target release), I saw that according to a [README file](http://ftp.ensembl.org/pub/release-100/README), "ensembl\_archive\_" directory should contain "Data on historical Ensembl releases". So, I wondered if this directory contains a map between ids from different releases. However, without any extra info about column names, contents, file formats etc, I couldn't really figure out how to use those files. So, I had to rule this one out too.
  
  ```
  Index of /pub/release-100/mysql/ensembl_archive_100/
  ../
  CHECKSUMS                                          26-Mar-2020 14:31                 132
  ens_release.txt.gz                                 26-Mar-2020 14:31                1098
  ensembl_archive_100.sql.gz                         26-Mar-2020 14:31                 943
  release_species.txt.gz                             26-Mar-2020 14:31               39620
  species.txt.gz                                     26-Mar-2020 14:31                6669
  
  ```
  
  Of note, I saw a similar "mysql" directory over at the directory of the release whose ids I want to convert (source release): `http://ftp.ensembl.org/pub/grch37/release-100/mysql/homo_sapiens_core_100_37/` . However, again, I do not understand its contents without any extra information.
  
  So, finally, I am not sure what is the best way to programmatically convert Ensembl ids from between different releases. As a test case scenario, I want to convert all the protein\_coding transcript ids from GRCh37 (release 75) to GRCh38 (release 100). If anybody has any suggestions about this, please kindly let me know.
  
  **Tags:** gene, ensembl, identifiers, conversion
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2021-03-22. Score: 1
  
  The ID history converter is available as a [Perl script](https://github.com/Ensembl/ensembl-tools/tree/release/103/scripts/id_history_converter).
  
  This script accesses these [MySQL tables](http://www.ensembl.org/info/docs/api/core/core_schema.html#gene_archive), so you could query these tables directly.
  
  Note that the mapping session is only one release at a time, so you would need to run 25 queries to go from release 75 to 100.
  
  > **Comments:**
  > **username_0**: Thank you for your answer. To use the perl script, I wonder if there is any tutorial to refer to. I am particularly interested in knowing how to "tell" the script to convert from a given specific release to other specific release e.g. from 75 to 76.  
  > **username_1**: It just gives you the whole history, you would need to parse out the releases you want.  
  > **username_0**: `IDmapper.pl` does the job, but I wish there was an easy way to install its dependency called `Bio::EnsEMBL::Registry`. I feel that perhaps a wrapper of sorts, for example in the Anaconda system, would be immensely helpful. (Please somebody make it happen! :D )  
  
  ---
  
  ### Answer by username_2 on 2021-03-25. Score: 0
  
  Here's one way to do it programmatically using the biomaRt package (your potential solution #1):
  
  ```
  library(biomaRt)
  library(dplyr)
  
  mart37 <- useMart(biomart = "ENSEMBL_MART_ENSEMBL",
                    host    = "grch37.ensembl.org",
                    path    = "/biomart/martservice",
                    dataset = "hsapiens_gene_ensembl")
  
  mart38 <- useMart(biomart = "ENSEMBL_MART_ENSEMBL",
                    path    = "/biomart/martservice",
                    dataset = "hsapiens_gene_ensembl")
  
  #pick all genes on chr22 from build 37 as a test dataset::
  out37 <- getBM(attributes = c('hgnc_symbol', 'chromosome_name',
                                'ensembl_gene_id', 'ensembl_transcript_id', 'ccds'),
                    filters = c('chromosome_name'),
                    values = list("22"),
                    mart = mart37) %>% #empty string CCDS ID's
    filter(ccds != "")
  
  #NOTE:: to check which filters can be used to query mart37, run this command:
  #View(listFilters(mart37))
  
  out38 <- getBM(attributes = c('hgnc_symbol', 'chromosome_name',
                                'ensembl_gene_id', 'ensembl_transcript_id', 'ccds'),
                          filters = c('ccds'),
                          values = list(out37$ccds),
                          mart = mart38)
  
  ```
  
  What this does is query the build 38 DB via the CCDS ID's of build 37. You can include other variables in this `out38` dataframe by simply adding to the attributes argument above. And in order to see all of the potential variables that can be included, you just have to run `View(listFilters(mart37))` or `View(listFilters(mart38))`, depending on which build you are interested in.
  
  Does this solve your problem?
  
  > **Comments:**
  > **username_0**: your solution could indeed be useful. I am just curious which ensembl release numbers are used when `mart37` and `mart38` were built. Is there a way to specify that?  
  > **username_2**: `mart37` corresponds to the GRCh37 release, while `mart38` corresponds to GRCh38. This is specified in the `host=` argument in the code above. `mart38` is the default, and so is not specified. Does this answer your question?  
  > **username_2**: P.S. if you're referring to the version number of the ensembl release, you can obtain this by adding `'ensembl_gene_id_version'` to the `attributes` array in the code above.  
  > **username_0**: By Ensembl release number I meant [version numbers of the assemblies maintained by ensembl](https://www.ensembl.info/category/01-release/), e.g. release 75 corresponds to GRCh37 for human genome.  
  > **username_0**: `ensembl_gene_id_version` could be useful if it is possible to know to which release that belongs to. e.g. how would I know if `x.3` id belongs to genome release 90 or 91? To get this information, I would have to run 'Ensembl IDmapper' which I would like to avoid, because of the issue I mentioned in my question.  
  
  ---
  
  