@VitalyFedyunin Hi! Thanks for your reply. I noticed that with my current versions, I also get the error immediately when I set a PyTorch seed or after a few epochs without seed and val_workers>0. I also find that I get the error when I have bum_workers>0 and I try reading from a pickled file containing PyTorch tensors (even if i save as numpy arrays it errors). The error also occurs sometimes when I try writing a pickled file (I write only for local_rank=0 process, but read the pickled file from both GPU processes in the DDP)
The first 2 errors are resolved with the nightly build. However the third, where I read in PyTorch tensors from a pickled file and have val_workers > 0 still persists. With val_workers=0 there is no error. Any idea on how to resolve this would be great! Please find a few more details below.
I have a small dataset, so I load all the data in the __init__ of my Dataset class. I then save it on my disk using pickle so I can save on dataloading time when I run my code again. Now, since I have 2 GPUs, DDP in pytorch-lightning starts 2 processes and each of these processes start reading from the pickle file. Both the training data and validation data are being read from pickle files. Epoch 0 training happens successfully but as soon as its starts validation, it crashes with the below error.
The error I get is -
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.                                                         [17/1779]
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.                                                                                                                                  
Exception raised from insert_events at ../c10/cuda/CUDACachingAllocator.cpp:1243 (most recent call first):                                                                              
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x42 (0x2b74fab78a52 in /home/usr/pytorch_nightly/lib/python3.8/site-packages/torch/lib/libc10.so)                 
frame #1: <unknown function> + 0x1c8ce (0x2b74fa8fc8ce in /home/usr/pytorch_nightly/lib/python3.8/site-packages/torch/lib/libc10_cuda.so)                                          
frame #2: c10::cuda::CUDACachingAllocator::raw_delete(void*) + 0x1a2 (0x2b74fa8fcee2 in /home/usr/pytorch_nightly/lib/python3.8/site-packages/torch/lib/libc10_cuda.so)            
frame #3: c10::TensorImpl::release_resources() + 0x9c (0x2b74fab6205c in /home/usr/pytorch_nightly/lib/python3.8/site-packages/torch/lib/libc10.so)                                
frame #4: <unknown function> + 0x292b79 (0x2b749d542b79 in /home/usr/pytorch_nightly/lib/python3.8/site-packages/torch/lib/libtorch_python.so)                                     
frame #5: <unknown function> + 0xacd961 (0x2b749dd7d961 in /home/usr/pytorch_nightly/lib/python3.8/site-packages/torch/lib/libtorch_python.so)                                     
<omitting python frames>                                                                                                                                                                
                                                                                                                                                                                        
^C/home/usr/pytorch_nightly/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py:1047: UserWarning: Detected KeyboardInterrupt, attempting graceful shutdown...        
  rank_zero_warn("Detected KeyboardInterrupt, attempting graceful shutdown...")                                                                                                         
^CException ignored in: <function _MultiProcessingDataLoaderIter.__del__ at 0x2b74fc2c85e0>                                                                                             
Traceback (most recent call last):                                                                                                                                                      
  File "/home/usr/pytorch_nightly/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1328, in __del__                                                               
    self._shutdown_workers()                                                                                                                                                            
  File "/home/usr/pytorch_nightly/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1301, in _shutdown_workers                                                     
    w.join(timeout=_utils.MP_STATUS_CHECK_INTERVAL)                                                                                                                                     
  File "/cvmfs/server/easybuild/software/2020/avx2/Core/python/3.8.10/lib/python3.8/multiprocessing/process.py", line 149, in join                                       
    res = self._popen.wait(timeout)                                                                                                                                                     
  File "/cvmfs/server/easybuild/software/2020/avx2/Core/python/3.8.10/lib/python3.8/multiprocessing/popen_fork.py", line 44, in wait                                     
    if not wait([self.sentinel], timeout):                                                                                                                                              
  File "/cvmfs/server/easybuild/software/2020/avx2/Core/python/3.8.10/lib/python3.8/multiprocessing/connection.py", line 931, in wait                                    
    ready = selector.select(timeout)                                                                                                                                                    
  File "/cvmfs/server/easybuild/software/2020/avx2/Core/python/3.8.10/lib/python3.8/selectors.py", line 415, in select                                                   
    fd_event_list = self._selector.poll(timeout)                                                                                                                                        
KeyboardInterrupt:                            
^CTraceback (most recent call last):                                                                                                                                                    
  File "/home/usr/mydir/mymodel/run.py", line 22, in <module>            
    main()                                                                                                                                                                              
  File "/home/usr/mydir/mymodel/run.py", line 18, in main                
    return train(CFG)                                                                                                                                                                   
  File "/mydir/usr/mymodel/mymodelsub/train.py", line 110, in train                                                                                         
    trainer.fit(model,dm)                                                                                                                                                                 File "/home/usr/pytorch_nightly/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 553, in fit                                                              
    self._run(model)                                                                                                                                                                    
  File "/home/usr/pytorch_nightly/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 921, in _run
    self._post_dispatch()
  File "/home/usr/pytorch_nightly/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 976, in _post_dispatch
    self.accelerator.teardown()
  File "/home/usr/pytorch_nightly/lib/python3.8/site-packages/pytorch_lightning/accelerators/gpu.py", line 57, in teardown
    super().teardown()
  File "/home/usr/pytorch_nightly/lib/python3.8/site-packages/pytorch_lightning/accelerators/accelerator.py", line 157, in teardown
    self.training_type_plugin.teardown()
  File "/home/usr/pytorch_nightly/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/parallel.py", line 143, in teardown
    self.lightning_module.cpu()
  File "/home/usr/pytorch_nightly/lib/python3.8/site-packages/pytorch_lightning/core/mixins/device_dtype_mixin.py", line 136, in cpu
    return super().cpu()
  File "/home/usr/pytorch_nightly/lib/python3.8/site-packages/torch/nn/modules/module.py", line 673, in cpu
    return self._apply(lambda t: t.cpu())
  File "/home/usr/pytorch_nightly/lib/python3.8/site-packages/torch/nn/modules/module.py", line 533, in _apply
    module._apply(fn)
  File "/home/usr/pytorch_nightly/lib/python3.8/site-packages/torch/nn/modules/module.py", line 533, in _apply
    module._apply(fn)
  File "/home/usr/pytorch_nightly/lib/python3.8/site-packages/torch/nn/modules/module.py", line 533, in _apply
    module._apply(fn)