I had the same problem and found that this has been fixed recently https://github.com/pytorch/pytorch/commit/99b3f5cd705018c96b5559daa89aa71e66fc7e8a 53
Unfortunately using pytorch-nightly instead of 1.1.0 broke my code somewhere else.
Is there a list of breaking changes landing in 1.2 somewhere?