Gotcha.
Another observation. The original environment runs successfully when I use num_workers=0 after 2 days. I will launch a couple of trials more in other machines where I’m getting the same issue.


After updating to Pytorch 1.9.0, the code keeps breaking. The error message made me realize that I was setting the wrong variable CUDA_LAUNCH_BLOCK. With CUDA_LAUNCH_BLOCKING=1, the code hang up completely. It did not progress while I was sleeping 
Does anything come to your mind?


Collecting environment information...                                       
PyTorch version: 1.9.0                                           
Is debug build: False                                            
CUDA used to build PyTorch: 11.1                                 
ROCM used to build PyTorch: N/A                                  
                                                                 
OS: Ubuntu 18.04.5 LTS (x86_64)                                  
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0                                        
Clang version: Could not collect                                         
CMake version: version 3.10.2
Libc version: glibc-2.27

Python version: 3.8 (64-bit runtime)
Python platform: Linux-4.15.0-147-generic-x86_64-with-glibc2.10
Is CUDA available: True
CUDA runtime version: 10.1.168
GPU models and configuration: 
GPU 0: GeForce RTX 2080 Ti
GPU 1: GeForce RTX 2080 Ti
GPU 2: GeForce RTX 2080 Ti
GPU 3: GeForce RTX 2080 Ti
GPU 4: GeForce RTX 2080 Ti
GPU 5: GeForce RTX 2080 Ti
GPU 6: GeForce RTX 2080 Ti
GPU 7: GeForce RTX 2080 Ti

Nvidia driver version: 460.80
cuDNN version: /usr/lib/x86_64-linux-gnu/libcudnn.so.7.5.0
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.19.1
[pip3] pytorch-lightning==0.9.0
[pip3] torch==1.9.0
[pip3] torchvision==0.10.0
[conda] blas                      1.0                         mkl  
[conda] cudatoolkit               11.1.74              h6bb024c_0    nvidia
[conda] ffmpeg                    4.3                  hf484d3e_0    pytorch
[conda] mkl                       2020.2                      256  
[conda] mkl-service               2.3.0            py38he904b0f_0  
[conda] mkl_fft                   1.3.0            py38h54f3939_0  
[conda] mkl_random                1.1.1            py38h0573a6f_0  
[conda] numpy                     1.19.1           py38hbc911f0_0  
[conda] numpy-base                1.19.1           py38hfa32c7d_0  
[conda] pytorch                   1.9.0           py3.8_cuda11.1_cudnn8.0.5_0    pytorch
[conda] pytorch-lightning         0.9.0                    pypi_0    pypi
[conda] torchvision               0.10.0               py38_cu111    pytorch


Side question, if I install with pip. Which dependencies are linked from the system?
The code environment that I’m using mixes conda and pip requirements. conda itself recommends to be careful about that afair. Thus, I’m considering to just realize on pip and use conda as shelter.