I’m not familiar with the Windows task manager output, but it will most likely also show the cached memory (which is allocated by PyTorch and can be reused).
The overhead depends also on the layer type.
E.g. convolution layers have a small memory footprint regarding their parameters, but a larger memory usage regarding the output volumes.
This conv layer will just use 9 FP32 parameters, which is tiny:
conv = nn.Conv2d(1, 1, 3, 1, 1m bias=False).cuda()

However, if we feed an input of [10, 1, 224, 224] to this layer, the output will have the same shape as the input and thus we double the memory usage.
Usually the number of output channels might be even larger, so depending on your model architecture, the overhead is expected.