There was a bug when there was uninitialized memory in unused parts of the input gradient 99. This is still in PyTorch releases including 1.1. Thanks to @yqwangustc’s fix  the nightlies / master would work better.
Best regards
Thomas