BigBirDNA
Pretraining BigBird on DNA sequences. This provides a base model for downstream DNA sequence analysis tasks

2. Language
The model will be trained in DNA

3. Model
BigBird.

4. Datasets
All the available DNA sequences.
Possible links to publicly available datasets include:

www.ncbi.nlm.nih.gov/genbank/ 5
Others can be found on www.insdc.org 2



5. Training scripts

There is a Flax BigBird implementation at transformers/modeling_flax_big_bird.py at master · huggingface/transformers · GitHub 11

There is transformers/run_mlm_flax.py at master · huggingface/transformers · GitHub 3 for word masking which we may be able to tweak for bigbird
There is GitHub - jerryji1993/DNABERT: DNABERT: pre-trained Bidirectional Encoder Representations from Transformers model for DNA-language in genome 9
that can be used to copy/tweak downstream tasks


6. (Optional) Challenges

There will be quite a bit of preprocessing involved
There will be quite a lot of data involved (about half a TB compressed)
Besides NLP skills there may be some bioinformatics skills required


7. (Optional) Desired project outcome

Pretrain a transformer that can emulate or improve on the downstream tasks mentioned reads below


8. (Optional) Reads
The following links can be useful to better understand the project and
what has previously been done.

https://arxiv.org/pdf/2007.14062.pdf 14
https://www.biorxiv.org/content/10.1101/2020.09.17.301879v1.full 10