Hello,
As far as I am aware, the pruning functionality currently does not provide a speedup because we are multiplying masks but not removing any weights/operations. I thought that having some discussion on the most straightforward way to realize this would be useful but if this is not the right place to discuss this, feel free to remove the post.


The most straightforward way that I can think of would be to create an entirely new network by copying the non-pruned weights. This has several issues tho.
a) It is weird for unstructured methods when e.g. not entire convolutions are pruned. You could copy convolutions that are not entirely pruned but then you would have to re-introduce masks for those partially pruned weights. Little cumbersome but possible.
b) For certain structured approaches it is easy, removing an entire channel when the convolutional layers are fully connected is not a problem. However, I have found better results when you look at individual convolutions to turn off. This means that you don’t prune entire channels, you remove some input connections to a channel. How do you copy the weights now? You need some sort of ‘pointer’ so that if you have to copy say only 10 convolutions out of the 64 input convolutions they need to know to which channels they should connect.


Maybe a different way (I don’t know how to do this) that circumvents both of aformentioned problems is to skip operations when there is only zeroes involved instead of still performing them. It has to be more general than for only convolutions of course but the idea is that you would simply not perform convolutional operations when the weight matrix is fully masked. This would remove both aformentioned problems but I don’t know how to implement it and what kind of overhead it would introduce.


Again, if such a more brainstorming post is not really intended here I will remove it but I would be interested in hearing other peoples thoughts on how to realize practical speedups for pruning.
Kind regards,
Richard