Previously when I was using PyTorch 1.6 + CUDA 10.2 (on RTX 2070) I did not see this behavior. Now I am using PyTorch 1.9 + CUDA  11.1 (on RTX 3090) I am seeing this behavior.
This thread 5 suggests that it is expected behavior. Looks like the latest nightly build solves the problem (reference) 9 .