Thanks for the notebook! I wasn’t able to run it on Colab for unknown reasons (the runtime just stopped), but was able to reproduce the issue locally in PyTorch 1.10.0. It seems to be fixed in the current master/nightly and works as expected with 1.11.0.dev20211101+cu113.
I couldn’t quickly find a related PR for the issue, but will check a bit more what might have been the issue in 1.10.