Thanks for the update.
I cannot reproduce it with CUDA10.2.89, PyTorch 1.6, NVIDIA driver 450.51.06 and the creation of a CUDATensor creates a CUDA context of ~940MB on a V100-SXM2 32GB using the conda binaries.
Building from source using CUDA11.1 and a newer PyTorch version creates a context of  ~816MB.