Wang-zipeng:

23068672 ---- print(torch.cuda.memory_reserved())
17863680 ---- print(torch.cuda.memory_allocated())
after inputs = conv(inputs), only one conv consume about 2G GPU memory


That’s ~23MB.
Note that there are additional lazy initializations for things like cublas, curand, cudnn.