I had the same problem.
I think the reason because conda does not see torch location (when I do “conda list” there is no torch package, eventhough torchvison and torchaudio are there).
I tried to install torch again using pip - in stead of conda install, the problem disappears. I don’t know if it will cause any incompatibility, but it solved my “slow import torch” problem.