I have two GPUs on my machine, one RTX 2080Ti and one 1080Ti. Last night during an experiment on RTX 2080Ti the computation failed, like when you encounter a NaN, and I’ve been having weird issues with my RTX 2080Ti ever since.
My syslog is filled with:
[ 1653.030199] NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Warp Exception on (GPC 5, TPC 4, SM 1): Illegal Instruction Encoding
[ 1653.030203] NVRM: Xid (PCI:0000:01:00): 13, Graphics SM Global Exception on (GPC 5, TPC 4, SM 1): Multiple Warp Errors

The same model with the same input yields different output depending on the GPU! Even on a simple convolution operation, the outputs can differ as much as 1e37. It seems like the issue must be from the CUDA driver and the below layers.
I’m currently running Driver Version: 410.78 CUDA Version: 10.0 with PyTorch 1, tested with Py 2.7 and 3.7.
np.random.seed(args.seed)
random.seed(args.seed)
torch.manual_seed(args.seed)
torch.cuda.set_device(0)
torch.cuda.manual_seed(args.seed)
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False
torch.backends.cudnn.enabled = True

# op = Conv2d(3, 128, kernel_size=(9, 9), stride=(1, 1), padding=(4, 4))
# op.weight.norm() = 6.5080
# op.bias.norm() = 0.4507
# x.norm() = 326.2348
x_cpu = x.to('cpu')
y_cpu = op.to('cpu')(x_cpu)

x_cuda0 = x.to('cuda:0')
y_cuda0 = op.to('cuda:0')(x_cuda0).cpu()

x_cuda1 = x.to('cuda:1')
y_cuda1 = op.to('cuda:1')(x_cuda1).cpu()

(y_cpu-y_cuda0).max()
> 1.5438e+20
(y_cpu-y_cuda1).max()
> 9.5367e-07
y_cpu.norm()
> 16999.3672
y_cuda0.norm()
> inf
y_cuda1.norm() 
> 16999.3672

Has anyone ever encountered a similar issue? Any ideas on what to do next?!