image1400×829 22.5 KB

I get the message as below when I’m training WideResNet with CIFAR-10.
RuntimeError: Function 'LogSoftmaxBackward' returned nan values in its 0th output.

So I wonder what’s the problem and I found the input is having nan values using the codes below.
for epoch in range(epochs):
    for i, (X, y) in enumerate(train_data):
        X, y = X.cuda(self.gpu, non_blocking=True), y.cuda(self.gpu, non_blocking=True)
        for j in range(X.size(0)):
            if X[j].isnan().any():
                print(j, X[j])

But interesting thing is that if I index from dataset, it’s just fine (no nan value).
What’s the problem? I still cannot understand…
I’m using pytorch official docker (pytorch/pytorch:1.8.1-cuda11.1-cudnn8-devel) and GPU with A100-SXM4-40GB.