kees:

Pretraining RoBERTA on DNA sequences. This provides a base model for downstream DNA sequence analysis tasks


Sounds interesting @kees, I was infact working on something similar → a jax/flax implementation of ProteinBERT 4. Pretraining RoBERTa seems like a good strategy. I’m definitely interested.