Agree with @agemagician - this could be a very interesting project to use BigBird for!
We have FlaxBigBird: transformers/modeling_flax_big_bird.py at master · huggingface/transformers · GitHub 1 merged and the official BigBird paper also did some experiments on protein sequence modeling: [2007.14062] Big Bird: Transformers for Longer Sequences 2
I think one should be able to slightly tweak https://github.com/huggingface/transformers/blob/master/examples/flax/language-modeling/run_mlm_flax.py for it to work well with BigBird  Also pinging our “BigBird” expert @vasudevgupta