Hi,
The only line were you could see a change here is between the net.to(gpu) and net.cpu() right?
The allocator on your system might not release the memory to the system right away to improve speed. And it looks like it does this as you can see that moving your model back to the cpu use less memory than when you created it at the beginning.