Could you create a single torch.randn(1).cuda() tensor and check the memory usage via nvidia-smi?
If the ~350MB are visible, they are coming from the CUDA context and you cannot reduce it.
The size of the context depends on the CUDA version, used device etc.