Hello,
I think this is a better way to do it (mentioned as the beginning):
# zero-out all the gradients corresponding to the pruned connections
for l,p in enumerate(cnn.parameters()):
    pruned_inds = pruned_inds_by_layer[l]
    p.grad.data[pruned_inds] = 0.

Because I have the mask (zero-weights), so I can update only non zero weights. However, my weights continue updating. I will try other options as well