PyTorch will allocate memory from the large or small pool, which has defined page sizes, so the reserved memory might be larger than the exact bytes needed to store the tensor.
Your current description of the model doesn’t fit the reported memory via nvidia-smi, so could you post the model definition as well as the input shape?