How are you planning on storing the gradient and reassigning it back?
If you just hold it on the GPU, the same amount of memory will be used.
I think you should check out torch.utils.checkpoint 9, which can be used to trade compute for memory.