By skimming through both environments I think the main difference is:
Old environment:

pytorch=0.4.0=py36hdf912b8_0

cuda90=1.0=h6433d27_0 and cudatoolkit=9.0=h13b8566_0

cudnn=7.1.2=cuda9.0_0

New environment:

pytorch=1.4.0=py3.8_cuda10.1.243_cudnn7.6.3_0
cudatoolkit=10.1.243=h6bb024c_0

Besides an older PyTorch, CUDA, and cudnn version, there were also two cuda packages installed and I’m not sure how they interact with each other.
Also, did you add synchronizations to your code when profiling manually?
Since CUDA operations are asynchronous, you would have to synchronize via torch.cuda.synchronize() before starting and stopping the timer.