Hi!
I am moving tensors between the CPU and GPU memory with .to(device) and .cpu().
I found out that all tensor that get in or out of the nn.Linear layer are locked in GPU memory.
That is why I created my own Linear layer and I found out that if require_grad=False I get the expected  use of memory in the GPU.
If require_grad=True, then I cannot swap the tensors to CPU memory.
.cpu() or deleting the reference with del tensor_var have no impact if require_grad=True.
Can someone explain the background to this behaviour and maybe suggest a workaround to be able to use Pytorch’s autograd?
A huge thank you in advance for every answer!