Solved by ptrblck in post #6 
              
              
                I’m not familiar with the Windows task manager output, but it will most likely also show the cached memory (which is allocated by PyTorch and can be reused). 
The overhead depends also on the layer type. 
E.g. convolution layers have a small memory footprint regarding their parameters, but a larger …