you might find some pointers here
  
    
    
    Using transformers (BERT, RoBERTa) without embedding layer Research
  
  
    I’m looking to train a RoBERTa model on protein sequences, which is in many ways similar to normal nlp training, but in others quite different. 
In the language of proteins, I have 20 characters instead of the normal 26 characters used in english (it is 26 right? :D), so that is rather similar. The big difference is that you don’t really combine the characters in proteins to actual words, but rather just keep each character as a distinct token or class. 
Hence essentially my input to the transfo…