I’m actually working on an algorithm for this very thing. Currently, the algorithm only works on Linear and Conv2D layers. GRU is in the works. Additionally, it will be updated soon to include removing target neurons.
How neurons are targetted for splitting is by taking an average of the absolute value of the gradients during an epoch. Then any neurons that are above the cutoff will get split.
But what you are referring to is not neuron splitting, but neuron removal. When that update is released, it will target neurons with gradient movement below the threshold you set.
Here is the Github with working examples and tests showing how this performs vs non-splitting.

  
      
      github.com
  
  
    

CerebralSeed/Neural-Splitting-with-Dice-Roll-in-Pytorch 3


  This demonstrates a dynamic neural splitting module implemented in Pytorch and combined with a "best of n" starting parameters. 

  
  
    
    
  
  


Please note that you can use this code for personal or research projects freely. If for other purposes, such as commercial, please contact me for licensing and support/development.