Solved by albanD in post #4 
              
              
                Unfortunately pytorch cannot know about the “irrelevant” part and only uses the chain rule. And in the chaine rule, you get 0 * nan = nan. 
There are a few issues on github that discuss this problem: https://github.com/pytorch/pytorch/issues/15506 and https://github.com/pytorch/pytorch/issues/26799 …