Really great work @RichardWang!
Here’s btw. the discussion about the learning rate decay through layers:

  
      github.com/google-research/electra
  
  
    
  
	  
  

  
    
      The implementation of layerwise learning rate decay 29
    

    
      
        opened 
        
          
        
        Apr 30, 2020
      
      

        
          closed 
        
          
        
        May 19, 2020
      
        

      
        
          
          importpandas
        
      
    
  



  electra/model/optimization.py
 
 
 Lines 188 to 193
 in
 7911132
 
 
 
 
 
 
 for layer in range(n_layers): 
 
...