@VitalyFedyunin @tinan5 I got the same error after few epochs in the nightly version as well…Does not seem to be fixed…
terminate called after throwing an instance of 'c10::CUDAError'
  what():  CUDA error: initialization error
Exception raised from insert_events at ../c10/cuda/CUDACachingAllocator.cpp:1243 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x42 (0x2ab1c8b51a52 in /home/me/pytorch_nightly/lib/python3.8/site-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0x1c8ce (0x2ab1c88d58ce in /home/me/pytorch_nightly/lib/python3.8/site-packages/torch/lib/libc10_cuda.so)
frame #2: c10::cuda::CUDACachingAllocator::raw_delete(void*) + 0x1a2 (0x2ab1c88d5ee2 in /home/me/pytorch_nightly/lib/python3.8/site-packages/torch/lib/libc10_cuda.so)
frame #3: c10::TensorImpl::release_resources() + 0x9c (0x2ab1c8b3b05c in /home/me/pytorch_nightly/lib/python3.8/site-packages/torch/lib/libc10.so)
frame #4: <unknown function> + 0x292b79 (0x2ab16b51bb79 in /home/me/pytorch_nightly/lib/python3.8/site-packages/torch/lib/libtorch_python.so)
frame #5: <unknown function> + 0xacd961 (0x2ab16bd56961 in /home/me/pytorch_nightly/lib/python3.8/site-packages/torch/lib/libtorch_python.so)
<omitting python frames>

terminate called after throwing an instance of 'c10::CUDAError'
  what():  CUDA error: initialization error
Exception raised from insert_events at ../c10/cuda/CUDACachingAllocator.cpp:1243 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x42 (0x2ab1c8b51a52 in /home/me/pytorch_nightly/lib/python3.8/site-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0x1c8ce (0x2ab1c88d58ce in /home/me/pytorch_nightly/lib/python3.8/site-packages/torch/lib/libc10_cuda.so)
frame #2: c10::cuda::CUDACachingAllocator::raw_delete(void*) + 0x1a2 (0x2ab1c88d5ee2 in /home/me/pytorch_nightly/lib/python3.8/site-packages/torch/lib/libc10_cuda.so)
frame #3: c10::TensorImpl::release_resources() + 0x9c (0x2ab1c8b3b05c in /home/me/pytorch_nightly/lib/python3.8/site-packages/torch/lib/libc10.so)
frame #4: <unknown function> + 0x292b79 (0x2ab16b51bb79 in /home/me/pytorch_nightly/lib/python3.8/site-packages/torch/lib/libtorch_python.so)
frame #5: <unknown function> + 0xacd961 (0x2ab16bd56961 in /home/me/pytorch_nightly/lib/python3.8/site-packages/torch/lib/libtorch_python.so)
<omitting python frames>
frame #36: <unknown function> + 0x91d1 (0x2ab15dae61d1 in /Core/python/3.8.10/lib/python3.8/lib-dynload/_pickle.cpython-38-x86_64-linux-gnu.so)
frame #37: <unknown function> + 0x11821 (0x2ab15daee821 in /Core/python/3.8.10/lib/python3.8/lib-dynload/_pickle.cpython-38-x86_64-linux-gnu.so)
frame #38: <unknown function> + 0x1052d (0x2ab15daed52d in /Core/python/3.8.10/lib/python3.8/lib-dynload/_pickle.cpython-38-x86_64-linux-gnu.so)
frame #39: <unknown function> + 0x12263 (0x2ab15daef263 in /Core/python/3.8.10/lib/python3.8/lib-dynload/_pickle.cpython-38-x86_64-linux-gnu.so)
frame #40: <unknown function> + 0x113d1 (0x2ab15daee3d1 in /Core/python/3.8.10/lib/python3.8/lib-dynload/_pickle.cpython-38-x86_64-linux-gnu.so)
frame #41: <unknown function> + 0x10c62 (0x2ab15daedc62 in /Core/python/3.8.10/lib/python3.8/lib-dynload/_pickle.cpython-38-x86_64-linux-gnu.so)
frame #42: <unknown function> + 0x1052d (0x2ab15daed52d in /Core/python/3.8.10/lib/python3.8/lib-dynload/_pickle.cpython-38-x86_64-linux-gnu.so)
frame #43: <unknown function> + 0x15090 (0x2ab15daf2090 in /Core/python/3.8.10/lib/python3.8/lib-dynload/_pickle.cpython-38-x86_64-linux-gnu.so)
frame #61: <unknown function> + 0x7f27 (0x2ab15c357f27 in /cvmfs/server/gentoo/2020/lib64/libpthread.so.0)
frame #62: clone + 0x3f (0x2ab15c47087f in /cvmfs/server/gentoo/2020/lib64/libc.so.6)

Traceback (most recent call last):
  File "/home/me/pytorch_nightly/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 990, in _try_get_data
    data = self._data_queue.get(timeout=timeout)
  File "/Core/python/3.8.10/lib/python3.8/multiprocessing/queues.py", line 107, in get
    if not self._poll(timeout):
  File "/Core/python/3.8.10/lib/python3.8/multiprocessing/connection.py", line 257, in poll
    return self._poll(timeout)
  File "/Core/python/3.8.10/lib/python3.8/multiprocessing/connection.py", line 424, in _poll
    r = wait([self], timeout)
  File "/Core/python/3.8.10/lib/python3.8/multiprocessing/connection.py", line 931, in wait
    ready = selector.select(timeout)
  File "/Core/python/3.8.10/lib/python3.8/selectors.py", line 415, in select
    fd_event_list = self._selector.poll(timeout)
  File "/home/me/pytorch_nightly/lib/python3.8/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
    _error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 20907) is killed by signal: Aborted. 

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/me/pytorch_nightly/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1045, in _run_train
    self.fit_loop.run()
  File "/home/me/pytorch_nightly/lib/python3.8/site-packages/pytorch_lightning/loops/base.py", line 111, in run
    self.advance(*args, **kwargs)
  File "/home/me/pytorch_nightly/lib/python3.8/site-packages/pytorch_lightning/loops/fit_loop.py", line 200, in advance
    epoch_output = self.epoch_loop.run(train_dataloader)
  File "/home/me/pytorch_nightly/lib/python3.8/site-packages/pytorch_lightning/loops/base.py", line 112, in run
    self.on_advance_end()
  File "/home/me/pytorch_nightly/lib/python3.8/site-packages/pytorch_lightning/loops/epoch/training_epoch_loop.py", line 177, in on_advance_end
    self._run_validation()
  File "/home/me/pytorch_nightly/lib/python3.8/site-packages/pytorch_lightning/loops/epoch/training_epoch_loop.py", line 256, in _run_validation
    self.val_loop.run()
  File "/home/me/pytorch_nightly/lib/python3.8/site-packages/pytorch_lightning/loops/base.py", line 111, in run
    self.advance(*args, **kwargs)
  File "/home/me/pytorch_nightly/lib/python3.8/site-packages/pytorch_lightning/loops/dataloader/evaluation_loop.py", line 110, in advance
    dl_outputs = self.epoch_loop.run(
  File "/home/me/pytorch_nightly/lib/python3.8/site-packages/pytorch_lightning/loops/base.py", line 111, in run
    self.advance(*args, **kwargs)
  File "/home/me/pytorch_nightly/lib/python3.8/site-packages/pytorch_lightning/loops/epoch/evaluation_epoch_loop.py", line 93, in advance
    batch_idx, batch = next(dataloader_iter)
  File "/home/me/pytorch_nightly/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 521, in __next__
    data = self._next_data()
  File "/home/me/pytorch_nightly/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1186, in _next_data
    idx, data = self._get_data()
  File "/home/me/pytorch_nightly/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1152, in _get_data
    success, data = self._try_get_data()
  File "/home/me/pytorch_nightly/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1003, in _try_get_data
    raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) from e
RuntimeError: DataLoader worker (pid(s) 20907) exited unexpectedly

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/me/scratch/myproj/run.py", line 20, in <module>
    main()
  File "/home/me/scratch/myproj/run.py", line 16, in main
    return train(CFG)
  File "/scratch/me/myproj/myprojsub/train.py", line 111, in train
    trainer.fit(model,dm)
  File "/home/me/pytorch_nightly/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 553, in fit
    self._run(model)
  File "/home/me/pytorch_nightly/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 918, in _run
    self._dispatch()
  File "/home/me/pytorch_nightly/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 986, in _dispatch
    self.accelerator.start_training(self)
  File "/home/me/pytorch_nightly/lib/python3.8/site-packages/pytorch_lightning/accelerators/accelerator.py", line 92, in start_training
    self.training_type_plugin.start_training(trainer)
  File "/home/me/pytorch_nightly/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 161, in start_training
    self._results = trainer.run_stage()
  File "/home/me/pytorch_nightly/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 996, in run_stage
    return self._run_train()
  File "/home/me/pytorch_nightly/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1058, in _run_train
    self.training_type_plugin.reconciliate_processes(traceback.format_exc())
  File "/home/me/pytorch_nightly/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/ddp.py", line 451, in reconciliate_processes
    os.kill(pid, signal.SIGKILL)
ProcessLookupError: [Errno 3] No such process