Hi @tueboesen,
Yes, it will work. It can give you a very close results compared to MSA methods, sometimes even better results. If you combine it with MSA, it will even give you a better results compared to MSA methods alone.
We have trained (Transformer XL, XLNet, Bert, Albert, Electra and T5) for Uniref100 and BFD dataset. I would recommend to simply use on of these models, because it requires tremendous amount of computing power to reach good results.
You can find them here:

  
      
      github.com
  
  
    

agemagician/ProtTrans 34


  ProtTrans is providing state of the art pretrained language models for proteins. ProtTrans was trained on thousands of GPUs from Summit and hundreds of Google TPUs using Transformers Models. - agem...

  
  
    
    
  
  



  
      huggingface.co
  
  
    

Rostlab (Rostlab) 7



  
  
    
    
  
  


You can find more details on our paper:

  
      
      bioRxiv – 21 Jul 20
  
  
    

ProtTrans: Towards Cracking the Language of Life’s Code Through... 4

Computational biology and bioinformatics provide vast data gold-mines from protein sequences, ideal for Language Models (LMs) taken from Natural Language Processing (NLP). These LMs reach for new prediction frontiers at low inference costs. Here, we...


  
  
    
    
  
  


Facebook also trained Roberta using Unrief50 dataset:

  
      
      github.com
  
  
    

facebookresearch/esm 10


  Evolutionary Scale Modeling (esm): Pretrained language models for proteins

  
  
    
    
  
  


Unfortunately, we don’t have a notebook for training from scratch, but you can find more details to replicate our results here:

  
      github.com/agemagician/ProtTrans
  
  
    
  
	  
  

  
    
      Source code of the models 7
    

    
      
        opened 
        
          
        
        Oct 11, 2020
      
      

        
          closed 
        
          
        
        Oct 12, 2020
      
        

      
        
          
          hadim
        
      
    
  



  Do you have the source code of the various pre-trained models?



    question


  
  
    
    
  
  


@patrickvonplaten :
You meant :

  
      
      github.com
  
  
    

agemagician/ProtTrans 34


  ProtTrans is providing state of the art pretrained language models for proteins. ProtTrans was trained on thousands of GPUs from Summit and hundreds of Google TPUs using Transformers Models. - agem...

  
  
    
    
  
  


Not :

  
      
      github.com
  
  
    

agemagician/CodeTrans 2


  Pretrained Language Models for Source code. Contribute to agemagician/CodeTrans development by creating an account on GitHub.

  
  
    
    
  
  



ProtTrans: Provides the SOT pre-trained models for protein sequences.
CodeTrans: Provides the SOTpre-trained models for computer source code.