I was doing a GPU memory usage study for deployment and I noticed by default PyTorch is blocking out ~800mb of VRAM. I thought it might be model specific so I tried moving a Linear(1,1) layer to gpu and it took the same amount of memory, but moving another Linear(1,1) didn’t take any memory.
happens in 1.4 and 1.5