I don’t know what the expected memory usage in PyTorch on the host would be at the moment, but did you measure it after just importing PyTorch? I would expect the import to preload some data as well as 3rd party libs (e.g. MKL), which would take some space (but it might be way lower than 1GB).
In case you are sure the memory is used by your model, you could check if any outputs, optimizers etc. are still stored, which could reference the parameters and/or activations.