Do you initialize self.conv somewhere, as I cannot find it.
If you use torch.Tensor, the values will be uninitialized, thus they might contain any values including NaN.
Could you try to use torch.randn or a specific initialization for your conv kernels and try your code again?