Hey,
I’d recommend taking a look at this repo: https://github.com/agemagician/CodeTrans 88 by @agemagician . This repo uses transformer models for protein sequences if I understand it correctly.
Also, taking a look at those models:

  
      huggingface.co
  
  
    

Rostlab (Rostlab) 26



  
  
    
    
  
  


might help. Not sure if there is a notebook on doing protein sequence LM, maybe @agemagician has a good pointer by chance