Are you constantly seeing the ~40x increase in each odd iteration?

yes

If so, could you update to 1.9.0 and rerun the code, as I’m having trouble reproducing the issue?



1.9.0+cu111: both versions are fast (same behavior you observed)

1.8.1+cu111: here I have the reported performance gap

So, it seems the venv must contain the PyTorch LTS version to see this issue, which I installed via:
pip3 install torch==1.8.1+cu111 torchvision==0.9.1+cu111 torchaudio==0.8.1 -f https://download.pytorch.org/whl/lts/1.8/torch_lts.html