@ptrblck: I’ve checked that issue and tested everything listed there, but increasing the shared memory did not solve this issue.
@justusschock: So as it turns out, when I put my complete script into a main()-handle (which I thought I had already tried but apparently not :)) I do not get a full system freeze any longer! However, scripts similarly stop running, with either one of these errors being thrown:


RuntimeError: cuda runtime error (4) : unspecified launch failure at /opt/conda/conda-bld/pytorch_1518243271935/work/torch/lib/THC/generated/../generic/THCTensorMathPointwise.cu:630 during optimizer.step().

DataLoader worker (pid 15189) is killed by signal: Segmentation fault or Illegal Instruction which happens arbitrarily, also either during optimizer.step().

Full errors are appended.
Any ideas what could cause this/what generally leads to errors like this?
— FULL ERRORS —
EXAMPLE 1
ERROR: Unexpected segmentation fault encountered in worker
Traceback (most recent call last):
  File "script.py", line 590, in <module>
    main()
  File "script.py", line 540, in main
    trainer(epoch)
  File "script.py", line 395, in trainer
    optimizer.step()
  File "/home/user/software/miniconda3/envs/D3L/lib/python3.6/site-packages/torch/optim/adam.py", line 72, in step
    denom = exp_avg_sq.sqrt().add_(group['eps'])
  File "/home/user/software/miniconda3/envs/D3L/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 175, in handler
    _error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 9437) is killed by signal: Segmentation fault.

EXAMPLE 2
THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1518243271935/work/torch/lib/THC/generic/THCTensorMathPairwise.cu line=81 error=4 : unspecified launch failure                                   | 418/2092 [02:38<10:34,  2.64it/s]
Traceback (most recent call last):
  File "script.py", line 590, in <module>
    main()
  File "script.py", line 540, in main
    trainer(epoch)
  File "script.py", line 395, in trainer
    optimizer.step()
  File "/home/user/software/miniconda3/envs/D3L/lib/python3.6/site-packages/torch/optim/adam.py", line 69, in step
    exp_avg.mul_(beta1).add_(1 - beta1, grad)
RuntimeError: cuda runtime error (4) : unspecified launch failure at /opt/conda/conda-bld/pytorch_1518243271935/work/torch/lib/THC/generic/THCTensorMathPairwise.cu:81

EXAMPLE 3
Traceback (most recent call last):
  File "script.py", line 590, in <module>
    main()
  File "script.py", line 540, in main
    trainer(epoch)
  File "script.py", line 395, in trainer
    optimizer.step()
  File "/home/user/software/miniconda3/envs/D3L/lib/python3.6/site-packages/torch/optim/adam.py", line 72, in step
    denom = exp_avg_sq.sqrt().add_(group['eps'])
  File "/home/user/software/miniconda3/envs/D3L/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 175, in handler
    _error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 6114) is killed by signal: Illegal instruction.