I’ve been having the same problem for the past 6 months! Pain in the ass to reproduce as it only happens every now and then. On CPU the tensors are fine but when sending them to GPU, some individual values may flip to nan.
I’ve never observed the problem in our 4 x 4 P100 server if I run my models on a single node but when training models on multiple nodes (SLURM and DPP) this problem occurs again. I’ve had talks about this problem with out server provider and we tried swapping GPUs but the problem persists.
pin_memory=False seems to not fix the problem for me. I’m happy to provide additional details about our system if it helps! Here’s the output of collect_env if it helps at all.
Collecting environment information...
PyTorch version: 1.8.1+cu102
Is debug build: False
CUDA used to build PyTorch: 10.2
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.2 LTS (x86_64)
GCC version: (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
Clang version: Could not collect
CMake version: Could not collect

Python version: 3.8 (64-bit runtime)
Is CUDA available: True
CUDA runtime version: Could not collect
GPU models and configuration: 
GPU 0: Tesla P100-PCIE-16GB
GPU 1: Tesla P100-PCIE-16GB
GPU 2: Tesla P100-PCIE-16GB
GPU 3: Tesla P100-PCIE-16GB

Nvidia driver version: 450.102.04
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.20.2
[pip3] pytorch-lightning==1.3.0
[pip3] torch==1.8.1
[pip3] torchmetrics==0.2.0
[pip3] torchvision==0.9.1
[conda] _pytorch_select           0.1                       cpu_0  
[conda] blas                      1.0                         mkl  
[conda] cudatoolkit               11.0.221             h6bb024c_0  
[conda] libmklml                  2019.0.5                      0  
[conda] mkl                       2020.2                      256  
[conda] mkl-service               2.3.0            py38he904b0f_0  
[conda] mkl_fft                   1.3.0            py38h54f3939_0  
[conda] mkl_random                1.1.1            py38h0573a6f_0  
[conda] numpy                     1.20.2                   pypi_0    pypi
[conda] numpy-base                1.19.2           py38hfa32c7d_0  
[conda] pytorch-lightning         1.3.0                    pypi_0    pypi
[conda] torch                     1.8.1                    pypi_0    pypi
[conda] torchmetrics              0.2.0                    pypi_0    pypi
[conda] torchvision               0.9.1                    pypi_0    pypi