The question is tricky. First the paper 2 is suspicious… At least for me.
I just fast-forward it. They seam to optimize the computation to reduce the heat…
Definitely, not the memory because the shape of the tensors will be the same.
The original idea of the parameters is they should not be read only.
If you freeze weights then how can you learn?
If huge amount of your weight values are 0 you haven’t initialized your layer well, or mean and std are not normalized going through the layers.
I would ignore this paper.
If from some reason you would still like to deal with the paper (say you would like to implement it):


Try using model.parameters() instead of model.named_parametrers(), since named parameters are subset of parameters.


.numpy() would convert tensor to numpy array, so don’t set that to a tensor. In your case tensor and grad_tensor should be tensors with the same shape.


use torch.from_numpy(arr) if you have numpy arrays and you would like to convert them to tensors.


Although CUDA code runs on both CPU and GPU, I would not hardcode .cpu() or .gpu(), instead always use .to(device), unless you are positive you will only run your code on GPU. To detect the device you can use this:


device = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu')