There are some out-of-the-box pruning functionalities directly in PyTorch now, starting from v1.4, that will handle this for you, for a subset of pruning techniques. You can find them under  torch.nn.utils.prune .