I have the following minimal example:
import torch

torch.manual_seed(137)

L = torch.nn.Linear(8, 1)
x = torch.cat((torch.rand((4, 8)), torch.full((4,8), float('nan'))), dim=0)
x = L(x)[:4].sum()
x.backward()

print(L.weight.grad)

The gradients are NaN, although the output does not depend on the NaN entries.
Is this behaviour expected?
(In more complex architectures the same behaviour can be achieved by setting the unused entries to values that are close to overflowing the 32bit floats.)