Hi, Gianluca.
It should be possible with some modification.
I created an issue to detail how to do it.
  

      github.com/richarddwang/electra_pytorch
  

  
    
  
	  
  

  
    
      How can I pretrain ELECTRA starting from weights from google ? 4
    

    
      
        opened 
        
          
        
        May 15, 2021
      
      

        
          closed 
        
          
        
        May 15, 2021
      
        

      
        
          
          richarddwang
        
      
    

    
    
  


  
    This issue is to answer the question from hugggingface forum.

Although I have…n't tried it, it should be possible.

1. Make sure `my_model` is set to `False` to use huggingface model
https://github.com/richarddwang/electra_pytorch/blob/ab29d03e69c6fb37df238e653c8d1a81240e3dd6/pretrain.py#L43

2. Change `model(config)` -> `model.from_pretrained(model_name)`
https://github.com/richarddwang/electra_pytorch/blob/ab29d03e69c6fb37df238e653c8d1a81240e3dd6/pretrain.py#L364-L365

3. Be careful about `size`, `max_length`, and other configs
https://github.com/richarddwang/electra_pytorch/blob/ab29d03e69c6fb37df238e653c8d1a81240e3dd6/pretrain.py#L38
https://github.com/richarddwang/electra_pytorch/blob/ab29d03e69c6fb37df238e653c8d1a81240e3dd6/pretrain.py#L76-L81

Note: ELECTRA models published are actually ++ model described in appendix D, and max sequence length of ELECTRA-Small/ Small++ is 128/512.

Feel free to tag me if you have other questions.