Answers.Body: <p>Dfam has recently launched a sister resource, <a href="http://www.dfam-consensus.org/" rel="noreferrer"><strong>Dfam_consensus</strong></a>, whose <a href="https://xfam.wordpress.com/2017/05/18/introducing-dfam_consensus-dfams-consensus-sequence-twin/" rel="noreferrer">stated aim</a> is to replace RepBase. From the annoucement:</p>
  
  <blockquote>
    <p>Dfam_consensus provides an open framework for the community to store both seed alignments (multiple alignments of instances for a given family) and the corresponding consensus sequence model.</p>
  </blockquote>
  
  <p>Both RepeatMasker and RepeatModeler have been updated to support Dfam_consensus.</p>
  
  <p>I haven’t tried it yet but it looks promising.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-06-01T12:15:47.873Z
Answers.LastActivityDate: 2017-06-01T12:15:47.873Z
Answers.UserDisplayName: Konrad Rudolph
Answers.Body: <p>You could use <a href="http://bix.ucsd.edu/repeatscout/" rel="noreferrer">RepeatScout</a>, which has defined repeat libraries for a limited number of species (including human, mouse, and rat). If your taxon is not represented, you can also do de novo repeat prediction with RepeatScout to build your own library to feed to RepeatMasker. The <a href="http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.517.8206&amp;rep=rep1&amp;type=pdf" rel="noreferrer">RepeatScout publication</a> includes some comparisons with RepBase. Another related tool is <a href="http://www.repeatmasker.org/RepeatModeler/" rel="noreferrer">RepeatModeler</a>, which wraps RepeatScout with <a href="http://eddylab.org/software/recon/" rel="noreferrer">RECON</a> and some other programs, and shares authors with the RepeatMasker team.</p>
  
  <p>On the plus side RepeatScout/RepeatModeler are <a href="https://github.com/mmcco/RepeatScout" rel="noreferrer">open source</a> and do not use manual curation, meeting your criteria. On the negative, I'm not sure exactly how RepeatModeler and the component tools are maintained. The RepeatScout web and github pages have not been updated for several years, although the RepeatModeler page shows its latest release was in 2017. Anyway, I know that some combination of RepeatScout/RepeatModeler have been used to annotate repeats for some <em>fairly recent</em> newly sequenced genomes, e.g. for <a href="https://www.nature.com/nature/journal/v513/n7518/full/nature13726.html" rel="noreferrer">cichlids</a>, <a href="https://www.nature.com/nature/journal/v496/n7445/full/nature12027.html" rel="noreferrer">coelacanth</a>, and <a href="https://bmcgenomics.biomedcentral.com/articles/10.1186/1471-2164-14-95" rel="noreferrer">Darwin's finch</a>, so I think it's fair to say this kind of approach is accepted in the field, at least for vertebrate genome projects.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-06-01T12:09:58.040Z
Answers.LastActivityDate: 2017-06-01T12:09:58.040Z
Answers.UserDisplayName: Chris_Rands
Answers.Body: <p>AFAIK Dfam and Repbase are currently the two best sources of (a variety of) TE sequences.</p>
  
  <p>In my genome annotations I have used RepeatModeler+RepeatMasker and then later used Repbase+tblastx and Dfam+nhmmer to classify them.</p>
  
  <p>The classification process in my pipeline PhyLTR (<a href="https://github.com/mcsimenc/PhyLTR" rel="nofollow noreferrer">https://github.com/mcsimenc/PhyLTR</a>) is based on Dfam and Repbase. The process I used for LTR identification is</p>
  
  <ol>
  <li>Putative ID with LTRHarvest (based on structural sequence characteristics)</li>
  <li>Classification-by-homology to Repbase and Dfam</li>
  <li>Removal of elements without homology to sequences in Repbase or Dfam.</li>
  </ol>
  
  <p>This results in a set of LTR-Rs which are full-length and have evidence that they are LTR-Rs.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-27T16:17:13.620Z
Answers.LastActivityDate: 2019-02-27T18:42:54.970Z
Answers.LastEditDate: 2019-02-27T18:42:54.970Z
Answers.UserDisplayName: tomato_soup
Answers.Body: <p>+1 for taking issue with RepBase.</p>
  
  <p>I use the annotations from <a href="http://hammelllab.labsites.cshl.edu/software/" rel="nofollow noreferrer">the Hammell Lab GTFs that they put out with TEtoolkit</a>. It is similar to what you described to be using so this may be a redundant and useless answer but from the digging I’ve done they do seem to be comprehensive and well curated (for Drosophila, at least). </p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-12-14T06:52:14.230Z
Answers.LastActivityDate: 2017-12-14T06:52:14.230Z
Answers.UserDisplayName: AntiSocialBehaviourOrder
Answers.Body: <p>For pre-existing reliabe TE libraries it is a bit of a mess, because not everybody deposits the species-specific TE libraries to a database like RepBase. And as far as I know DFAM contains only human resources, or am I wrong?</p>
  
  <p>As for <em>de novo</em> generation of species-specific TE libraries (which should be done for any species not already present in eg. RepBase):
  There is no "gold-standard" how to tackle this best.
  In principle one has to think about two main parts
  -repeat detection
  -annotation</p>
  
  <p>For <strong>repeat detection</strong> I would recommend using a combination of two things (which is necessary, because TE copies might miss in the assemblies as repetitive regions tend to be difficult to assemble and thrown away in the final assembly).</p>
  
  <p>I) Repeat detection from raw reads (as with e.g. DNApipeTE or tedna or RepeatExplorer). For me, DNAPipeTE worked quite nicely, but everything has pros and cons.
  II) Repeat detection from assemblies (as with e.g. REPET or as mentioned before RepeatModeler)</p>
  
  <p>Then the <strong>annotation</strong> of these repeats is tricky too, because most methods are relying on homology between the <em>de novo</em> TEs and the TEs from some (probably distantly) related species. But some programs take also structure into account (like REPCLASS). REPET can do both detection and annotation, but is a pain to get to run. </p>
  
  <p>I would recommend using some programs to do de novo repeat detection on your species of interest on both the raw reads and assembly, clustering these libraries together (with e.g. uclust and 95% identity) and then run an annotation with homology and structural identification.</p>
  
  <p>Probably the programs will <strong>not give you complete, full-length TE</strong>s but rather consensus sequences of several copies from TE families.
  If you want, you could search all copies of one family, extract them from the contigs plus boundaries and align them manually and curate boundaries manually. Then extend boundaries if not hitting the surrounding (non-alignable) regions or landmarks of TEs like LTRs or TIRs or so. But this is very time consuming if you only want to compare TE abundance between species for example, I would not do this and rather compare the abundance using read coverage (as in <a href="https://doi.org/10.1093/molbev/msv261" rel="noreferrer">Bast et al. 2016</a>). Depends all on the questions you want to ask.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-06-27T08:06:22.357Z
Answers.LastActivityDate: 2017-06-27T08:06:22.357Z
Answers.UserDisplayName: Jens Bast
Answers.Body: <p>I know this question is a bit old, but this is still an issue for a lot of researchers not being able to access RepBase. It seems now that the most recent version of RepeatMasker is dependent upon RepBase for full functionality if masking anything other than human (currently DFAM only has human models). I recently discovered a <em>de novo</em> repeat masking approach called REpeat Detector (Red).  This might be a solution for some looking to mask repeats on a genome assembly for annotation.  The paper is <a href="https://doi.org/10.1186/s12859-015-0654-5" rel="nofollow noreferrer">here</a>. I also then wrote a wrapper around Red to make it a bit more easy to soft-mask a genome, which you can find <a href="https://github.com/nextgenusfs/redmask" rel="nofollow noreferrer">here</a>.  </p>
  
  <p>One of the limitations with Red is that the repeats are not classified, so they are only identified. You would have to use some of the other tools mentioned above to try to classify them. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-06-25T16:59:08.893Z
Answers.LastActivityDate: 2018-06-26T10:15:32.810Z
Answers.LastEditDate: 2018-06-26T10:15:32.810Z
Answers.UserDisplayName: jpalmer
Body: <p>I’m using the <a href="http://www.girinst.org/server/RepBase/index.php" rel="noreferrer">RepBase libraries</a> in conjunction with RepeatMasker to get genome-wide repeat element annotations, in particular for transposable elements.</p>
  
  <p>This works well enough, and seems to be the de facto standard in the field.</p>
  
  <p>However, there are two issues with the use of RepBase, which is why I (and others) have been looking for alternatives (so far without success):</p>
  
  <ol>
  <li><p><a href="http://www.girinst.org/accountservices/register.php?commercial=0" rel="noreferrer">RepBase isn’t open data</a>. Their academic license agreement includes a clause that <em>explicitly forbids dissemination of data derived from RepBase</em>. It’s unclear to what extent this is binding/enforceable, but it effectively prevents publishing at least some of the data I’m using and generating. This is unacceptable for <a href="https://en.wikipedia.org/wiki/Open_science" rel="noreferrer">open science</a>.</p>
  
  <ul>
  <li>Subordinate to this, the subscription model of RepBase also makes it impossible to integrate RepBase into fully automated pipelines, because user interaction is required to subscribe to RepBase, and to provide the login credentials.</li>
  </ul></li>
  <li><p>RepBase is heavily manually curated. This is both good and bad. Good, because manual curation of sequence data is often the most reliable form of curation. On the flip side, manual curation is inherently biased; and worse, it’s hard to quantify this bias — <a href="https://bmcbioinformatics.biomedcentral.com/articles/10.1186/1471-2105-7-474" rel="noreferrer">this is acknowledged by the RepBase maintainers</a>.</p></li>
  </ol>
  
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-06-01T12:29:46.460Z
Comments.Text: Were you only asking about defined repeat libraries? I interpreted it slightly more broadly as about tools used to build the libraries also (which becomes relevant when genomes from new taxa are sequenced)
Comments.UserDisplayName: Chris_Rands
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-06-01T12:35:31.297Z
Comments.Text: @Chris_Rands Both (libraries and tools). Your answer is spot-on.
Comments.UserDisplayName: Konrad Rudolph
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-06-01T14:19:25.843Z
Comments.Text: The goal is to build the annotated library of repetitions, or to mask repetitive parts of a genome?
Comments.UserDisplayName: Kamil S Jaron
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-06-01T14:24:02.673Z
Comments.Text: @KamilSJaron I’m working with TEs, so I need the annotated library, not (merely) a repeat masked sequence.
Comments.UserDisplayName: Konrad Rudolph
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-06-01T14:32:13.727Z
Comments.Text: Ouch, but for TEs just a subset of repetitive regions, there are specialised tools to annotate them (like [DNApipeTE](https://github.com/clemgoub/dnaPipeTE) and [REPET](https://urgi.versailles.inra.fr/Tools/REPET)). Maybe you could specify it in the question.
Comments.UserDisplayName: Kamil S Jaron
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-06-01T14:34:07.237Z
Comments.Text: @KamilSJaron Nice, and this might be worth an answer. I’ll also update teh question. That said, I am also asking for repetitive elements beyond TEs.
Comments.UserDisplayName: Konrad Rudolph
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-06-02T14:47:33.773Z
Comments.Text: Also are you sure that RepBase is against publications that derive information based on their data? Could it be that they just don't want you sharing the raw data and files they provide. Since they have a good number of citations in different fields: https://scholar.google.de/scholar?um=1&ie=UTF-8&lr&cites=11574259945967474319
Comments.UserDisplayName: story
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-06-02T14:49:20.957Z
Comments.Text: @story They literally say so in the academic user agreement that I link to. Here’s the relevant quote: “You agree NOT to make the Repbase (or any part thereof, including Repbase Reports, Repeat Maps **and other derived materials, modified or not**) available to anyone outside your research group.” Emphasis mine. In fact, another clause in the agreement technically even forbids me from signing it because my institute requires public data deposition, so I’m probably not allowed to sign such agreements.”
Comments.UserDisplayName: Konrad Rudolph
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-06-02T14:53:10.500Z
Comments.Text: Ya that seems to agree with my previous statement. I guess my point is what exactly did you need to share (based on your original post) that would be considered from their database? I feel like this wouldn't include counts of features but sequences might be an issue.
Comments.UserDisplayName: story
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-06-02T14:55:32.033Z
Comments.Text: @story I need to potentially share all data that was used/generated in my analysis. This particularly includes the specific repeat annotation I used, which are derived from RepBase, as well as potentially sequence data from these repeats.
Comments.UserDisplayName: Konrad Rudolph
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-11-08T21:20:53.567Z
Comments.Text: This might be an old question, but someone is trying to set-up a new, open, alternative to repBase (which i snow going full commercial), or at least that is how I perceive it: https://twitter.com/TransposableMan/status/1060519887897067521
Comments.UserDisplayName: fridaymeetssunday
Tags: <sequence-annotation><database><repeat-elements><transposable-elements>
Title: Are there any RepBase alternatives for genome-wide repeat element annotations?
ThreadText: # Are there any RepBase alternatives for genome-wide repeat element annotations?
  
  (Asked by: username_0 on 2017-06-01)
  
  I’m using the [RepBase libraries](http://www.girinst.org/server/RepBase/index.php) in conjunction with RepeatMasker to get genome-wide repeat element annotations, in particular for transposable elements.
  
  This works well enough, and seems to be the de facto standard in the field.
  
  However, there are two issues with the use of RepBase, which is why I (and others) have been looking for alternatives (so far without success):
  
  1. [RepBase isn’t open data](http://www.girinst.org/accountservices/register.php?commercial=0). Their academic license agreement includes a clause that *explicitly forbids dissemination of data derived from RepBase*. It’s unclear to what extent this is binding/enforceable, but it effectively prevents publishing at least some of the data I’m using and generating. This is unacceptable for [open science](https://en.wikipedia.org/wiki/Open_science).
  
     * Subordinate to this, the subscription model of RepBase also makes it impossible to integrate RepBase into fully automated pipelines, because user interaction is required to subscribe to RepBase, and to provide the login credentials.
  2. RepBase is heavily manually curated. This is both good and bad. Good, because manual curation of sequence data is often the most reliable form of curation. On the flip side, manual curation is inherently biased; and worse, it’s hard to quantify this bias — [this is acknowledged by the RepBase maintainers](https://bmcbioinformatics.biomedcentral.com/articles/10.1186/1471-2105-7-474).
  
  **Tags:** sequence-annotation, database, repeat-elements, transposable-elements
  
  > **Comments:**
  > **username_1**: Were you only asking about defined repeat libraries? I interpreted it slightly more broadly as about tools used to build the libraries also (which becomes relevant when genomes from new taxa are sequenced)  
  > **username_0**: @username_1 Both (libraries and tools). Your answer is spot-on.  
  > **username_2**: The goal is to build the annotated library of repetitions, or to mask repetitive parts of a genome?  
  > **username_0**: @username_2 I’m working with TEs, so I need the annotated library, not (merely) a repeat masked sequence.  
  > **username_2**: Ouch, but for TEs just a subset of repetitive regions, there are specialised tools to annotate them (like [DNApipeTE](https://github.com/clemgoub/dnaPipeTE) and [REPET](https://urgi.versailles.inra.fr/Tools/REPET)). Maybe you could specify it in the question.  
  > **username_0**: @username_2 Nice, and this might be worth an answer. I’ll also update teh question. That said, I am also asking for repetitive elements beyond TEs.  
  > **username_3**: Also are you sure that RepBase is against publications that derive information based on their data? Could it be that they just don't want you sharing the raw data and files they provide. Since they have a good number of citations in different fields: https://scholar.google.de/scholar?um=1&ie=UTF-8&lr&cites=11574259945967474319  
  > **username_0**: @username_3 They literally say so in the academic user agreement that I link to. Here’s the relevant quote: “You agree NOT to make the Repbase (or any part thereof, including Repbase Reports, Repeat Maps **and other derived materials, modified or not**) available to anyone outside your research group.” Emphasis mine. In fact, another clause in the agreement technically even forbids me from signing it because my institute requires public data deposition, so I’m probably not allowed to sign such agreements.”  
  > **username_3**: Ya that seems to agree with my previous statement. I guess my point is what exactly did you need to share (based on your original post) that would be considered from their database? I feel like this wouldn't include counts of features but sequences might be an issue.  
  > **username_0**: @username_3 I need to potentially share all data that was used/generated in my analysis. This particularly includes the specific repeat annotation I used, which are derived from RepBase, as well as potentially sequence data from these repeats.  
  > **username_4**: This might be an old question, but someone is trying to set-up a new, open, alternative to repBase (which i snow going full commercial), or at least that is how I perceive it: https://twitter.com/TransposableMan/status/1060519887897067521  
  
  ---
  
  ## 6 answers
  
  ### Answer by username_0 on 2017-06-01. Score: 14
  
  Dfam has recently launched a sister resource, [**Dfam\_consensus**](http://www.dfam-consensus.org/), whose [stated aim](https://xfam.wordpress.com/2017/05/18/introducing-dfam_consensus-dfams-consensus-sequence-twin/) is to replace RepBase. From the annoucement:
  
  > Dfam\_consensus provides an open framework for the community to store both seed alignments (multiple alignments of instances for a given family) and the corresponding consensus sequence model.
  
  Both RepeatMasker and RepeatModeler have been updated to support Dfam\_consensus.
  
  I haven’t tried it yet but it looks promising.
  
  
  ---
  
  ### Answer by username_7 on 2017-06-27. Score: 7
  
  For pre-existing reliabe TE libraries it is a bit of a mess, because not everybody deposits the species-specific TE libraries to a database like RepBase. And as far as I know DFAM contains only human resources, or am I wrong?
  
  As for *de novo* generation of species-specific TE libraries (which should be done for any species not already present in eg. RepBase):
  There is no "gold-standard" how to tackle this best.
  In principle one has to think about two main parts
  -repeat detection
  -annotation
  
  For **repeat detection** I would recommend using a combination of two things (which is necessary, because TE copies might miss in the assemblies as repetitive regions tend to be difficult to assemble and thrown away in the final assembly).
  
  I) Repeat detection from raw reads (as with e.g. DNApipeTE or tedna or RepeatExplorer). For me, DNAPipeTE worked quite nicely, but everything has pros and cons.
  II) Repeat detection from assemblies (as with e.g. REPET or as mentioned before RepeatModeler)
  
  Then the **annotation** of these repeats is tricky too, because most methods are relying on homology between the *de novo* TEs and the TEs from some (probably distantly) related species. But some programs take also structure into account (like REPCLASS). REPET can do both detection and annotation, but is a pain to get to run.
  
  I would recommend using some programs to do de novo repeat detection on your species of interest on both the raw reads and assembly, clustering these libraries together (with e.g. uclust and 95% identity) and then run an annotation with homology and structural identification.
  
  Probably the programs will **not give you complete, full-length TE**s but rather consensus sequences of several copies from TE families.
  If you want, you could search all copies of one family, extract them from the contigs plus boundaries and align them manually and curate boundaries manually. Then extend boundaries if not hitting the surrounding (non-alignable) regions or landmarks of TEs like LTRs or TIRs or so. But this is very time consuming if you only want to compare TE abundance between species for example, I would not do this and rather compare the abundance using read coverage (as in [username_7 et al. 2016](https://doi.org/10.1093/molbev/msv261)). Depends all on the questions you want to ask.
  
  
  ---
  
  ### Answer by username_1 on 2017-06-01. Score: 5
  
  You could use [RepeatScout](http://bix.ucsd.edu/repeatscout/), which has defined repeat libraries for a limited number of species (including human, mouse, and rat). If your taxon is not represented, you can also do de novo repeat prediction with RepeatScout to build your own library to feed to RepeatMasker. The [RepeatScout publication](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.517.8206&rep=rep1&type=pdf) includes some comparisons with RepBase. Another related tool is [RepeatModeler](http://www.repeatmasker.org/RepeatModeler/), which wraps RepeatScout with [RECON](http://eddylab.org/software/recon/) and some other programs, and shares authors with the RepeatMasker team.
  
  On the plus side RepeatScout/RepeatModeler are [open source](https://github.com/mmcco/RepeatScout) and do not use manual curation, meeting your criteria. On the negative, I'm not sure exactly how RepeatModeler and the component tools are maintained. The RepeatScout web and github pages have not been updated for several years, although the RepeatModeler page shows its latest release was in 2017. Anyway, I know that some combination of RepeatScout/RepeatModeler have been used to annotate repeats for some *fairly recent* newly sequenced genomes, e.g. for [cichlids](https://www.nature.com/nature/journal/v513/n7518/full/nature13726.html), [coelacanth](https://www.nature.com/nature/journal/v496/n7445/full/nature12027.html), and [Darwin's finch](https://bmcgenomics.biomedcentral.com/articles/10.1186/1471-2164-14-95), so I think it's fair to say this kind of approach is accepted in the field, at least for vertebrate genome projects.
  
  
  ---
  
  ### Answer by username_5 on 2019-02-27. Score: 4
  
  AFAIK Dfam and Repbase are currently the two best sources of (a variety of) TE sequences.
  
  In my genome annotations I have used RepeatModeler+RepeatMasker and then later used Repbase+tblastx and Dfam+nhmmer to classify them.
  
  The classification process in my pipeline PhyLTR (<https://github.com/mcsimenc/PhyLTR>) is based on Dfam and Repbase. The process I used for LTR identification is
  
  1. Putative ID with LTRHarvest (based on structural sequence characteristics)
  2. Classification-by-homology to Repbase and Dfam
  3. Removal of elements without homology to sequences in Repbase or Dfam.
  
  This results in a set of LTR-Rs which are full-length and have evidence that they are LTR-Rs.
  
  
  ---
  
  ### Answer by username_6 on 2017-12-14. Score: 3
  
  +1 for taking issue with RepBase.
  
  I use the annotations from [the Hammell Lab GTFs that they put out with TEtoolkit](http://hammelllab.labsites.cshl.edu/software/). It is similar to what you described to be using so this may be a redundant and useless answer but from the digging I’ve done they do seem to be comprehensive and well curated (for Drosophila, at least).
  
  
  ---
  
  ### Answer by username_8 on 2018-06-25. Score: 2
  
  I know this question is a bit old, but this is still an issue for a lot of researchers not being able to access RepBase. It seems now that the most recent version of RepeatMasker is dependent upon RepBase for full functionality if masking anything other than human (currently DFAM only has human models). I recently discovered a *de novo* repeat masking approach called REpeat Detector (Red). This might be a solution for some looking to mask repeats on a genome assembly for annotation. The paper is [here](https://doi.org/10.1186/s12859-015-0654-5). I also then wrote a wrapper around Red to make it a bit more easy to soft-mask a genome, which you can find [here](https://github.com/nextgenusfs/redmask).
  
  One of the limitations with Red is that the repeats are not classified, so they are only identified. You would have to use some of the other tools mentioned above to try to classify them.
  
  
  ---
  
  