image1343×767 16.9 KB

for epoch in range(epochs):
    for i, (X, y) in enumerate(train_data):
        print(X.isnan().any())
        X, y = X.cuda(self.gpu, non_blocking=True), y.cuda(self.gpu, non_blocking=True)
        print(X.isnan().any())

I checked more and just found that loading to gpu makes this weird behaviors.
Am I using wrong way to load tensors to gpu? Or it’s related with hardware issues (gpu)?