Unfortunately pytorch cannot know about the “irrelevant” part and only uses the chain rule. And in the chaine rule, you get 0 * nan = nan.
There are a few issues on github that discuss this problem: https://github.com/pytorch/pytorch/issues/15506 62 and https://github.com/pytorch/pytorch/issues/26799 36 for example.