It is because of the dropout.
  
      github.com/huggingface/transformers
  
  
    
  
	  
  

  
    
      [Causal Language Modeling] seems not as expected 9
    

    
      
        opened 
        
          
        
        Mar 6, 2021
      
      

        
          closed 
        
          
        
        Mar 12, 2021
      
        

      
        
          
          voidful
        
      
    
  



  Problem
Causal Models is only attended to the left context. Therefore causal models should not depend on the right tokens.
For example, The...