Well, after an extensive testing I couldn’t find any symptoms of non-idempotency. It doesn’t mean it doesn’t exist, I just couldn’t reproduce it on my machine.
There was one sub-issue of codecov generating an invalid report when it fails to find a code coverage report for the “base” it checks against. When this happens it goes looking for the nearest hash with the coverage report, which often leads to a report that is not representative of the true impact of the proposed PR, since it is no longer comparing the proposed code changes to the base against which it’d be applied.
A fix that prevents generation of invalid reports for when the base is lacking has been applied:

  
      github.com/huggingface/transformers
  
  
    
  
    
  

  
    
      fix incorrect codecov reports 1
    

    
      huggingface:master ← stas00:patch-1
    

    
      
        opened 
        
          
        
        Aug 18, 2020
      
      

      
        
          
          stas00
        
      

      
        
          +4
          -0
        
      
    

  


  
  
    
    
  
  


But looking at the recent PRs with no code change, the problem is still there. We still get coverage changes reported, when there are none:
e.g. https://codecov.io/gh/huggingface/transformers/pull/6650/changes, https://codecov.io/gh/huggingface/transformers/pull/6650/changes