At least 800MiB of GPU memory will be used for PyTorch’s native GPU kernels (happens when you call .cuda() on a tensor or layer with parameters). Then when you use a cuBLAS kernel for the first time (think matrix multiply on GPU), a hundred or so MiB will be used up by the cuBLAS libraries. A similar thing happens with cuDNN when kernels like convolution are called. By the time most of the kernels in a typical model are covered, easily over 1GiB of device memory will be used by code alone.
In summary, the GPU libraries containing the kernels used by PyTorch are fairly hefty, and they have to go somewhere (in this case device memory).