I am trying to save memory as much as possible.
The implementation actually comes from DeepSpeed. this line set torch.data to a torch.ones(1) tensor. DeepSpeed/partition_parameters.py at master · microsoft/DeepSpeed · GitHub 1
However, the parameters memory didn’t release correctly. And I found out that if a parameter Tensor has its corresponding intermediate results, the parameters memory won’t be released.
The parameter data will only be gathered when they are needed. Like gather in pre_sub_module_backward_function here:DeepSpeed/stage3.py at master · microsoft/DeepSpeed · GitHub
But If the parameters memory not release correctly, here gathering won’t consume any other memory. In other word, I reset the param.data to parameter Tensor, but the memory_allocated  has no change.