This documentation page includes a command to prune particular heads of a model

  
      
      huggingface.co
  
  
    

Models — transformers 3.4.0 documentation 12



  
  
    
    
  
  


Bertviz visualisation tool might help to decide which heads to delete, see

  
      
      GitHub
  
  
    

jessevig/bertviz 5

Tool for visualizing attention in the Transformer model (BERT, GPT-2, Albert, XLNet, RoBERTa, CTRL, etc.) - jessevig/bertviz


  
  
    
    
  
  


As a quick guess, heads where the attention for a word is all to the same word, or is all to the subsequent word, and heads where all the attention from all words is to the SEP token, may not be very useful.
I guess you could also use the command to do some systematic pruning and test the results.