@stamatis,
thanks for the link, that’s interesting,I didn’t know people were pruning weights.
Have you seen this article about pruning heads

  
      
      arXiv.org
  
  
    

Are Sixteen Heads Really Better than One? 2

Attention is a powerful and ubiquitous mechanism for allowing neural models
to focus on particular salient pieces of information by taking their weighted
average when making predictions. In particular, multi-headed attention is a
driving force behind...