The “hang” you are seeing is most likely the JIT compilation for the (new) GPU architecture.
In PyTorch 1.1.0 CUDA>=11.0 wasn’t released yet, so you would need to cherry-pick all CUDA11 commits to this branch and recompile it.
I think the easier way would be to update the PyTorch code base and use the latest release (also for new bug fixes and new features).