Solved by CedricLy in post #2 
              
              
                Try out: 
grad_dic = {x[0]:x[1].data.grad for x in model.named_parameters()}

and save grad_dic somewhere. 
torch.save(grad_dic,<path>)
...
grad_dic = torch.load(<path>)