I tried switching the two GPUs: instead of
notebook#1 with GPU#0, notebook#2 with GPU#1,
I run
notebook#1 with GPU#1, notebook#2 with GPU#0,
And now I have the following error on the notebook#1, so the issue is clearly linked to the GPU. Instead of a stacktrace in the notebook, the notebook simply freezes and I see the error in the jupyter lab terminal (Having the error in the terminal instead of the notebook already happened with the other notebook. The exact error and where I get it – notebook/terminal – seems random, but it always happens in the notebook running with GPU#1):
terminate called after throwing an instance of 'c10::Error'
  what():  CUDA error: an illegal memory access was encountered (record at /pytorch/aten/src/ATen/cuda/CUDAEvent.h:118)
frame #0: std::function<std::string ()>::operator()() const + 0x11 (0x7fb65e532441 in /home/pinouchon/pycharms_remote/valx_trader/.venv/lib/python3.6/site-packages/torch/lib/libc10.so)
frame #1: c10::Error::Error(c10::SourceLocation, std::string const&) + 0x2a (0x7fb65e531d7a in /home/pinouchon/pycharms_remote/valx_trader/.venv/lib/python3.6/site-packages/torch/lib/libc10.so)
frame #2: <unknown function> + 0x12b2612 (0x7fb669fd8612 in /home/pinouchon/pycharms_remote/valx_trader/.venv/lib/python3.6/site-packages/torch/lib/libcaffe2_gpu.so)
frame #3: <unknown function> + 0x12a123a (0x7fb669fc723a in /home/pinouchon/pycharms_remote/valx_trader/.venv/lib/python3.6/site-packages/torch/lib/libcaffe2_gpu.so)
frame #4: <unknown function> + 0x12a5dda (0x7fb669fcbdda in /home/pinouchon/pycharms_remote/valx_trader/.venv/lib/python3.6/site-packages/torch/lib/libcaffe2_gpu.so)
frame #5: <unknown function> + 0x12a5ffe (0x7fb669fcbffe in /home/pinouchon/pycharms_remote/valx_trader/.venv/lib/python3.6/site-packages/torch/lib/libcaffe2_gpu.so)
frame #6: <unknown function> + 0x9aa446 (0x7fb65f0f5446 in /home/pinouchon/pycharms_remote/valx_trader/.venv/lib/python3.6/site-packages/torch/lib/libcaffe2.so)
frame #7: at::native::gru(at::Tensor const&, at::Tensor const&, c10::ArrayRef<at::Tensor>, bool, long, double, bool, bool, bool) + 0x10d (0x7fb65f0e93dd in /home/pinouchon/pycharms_remote/valx_trader/.venv/lib/python3.6/site-packages/torch/lib/libcaffe2.so)
frame #8: at::TypeDefault::gru(at::Tensor const&, at::Tensor const&, c10::ArrayRef<at::Tensor>, bool, long, double, bool, bool, bool) const + 0xc9 (0x7fb65f42ab19 in /home/pinouchon/pycharms_remote/valx_trader/.venv/lib/python3.6/site-packages/torch/lib/libcaffe2.so)
frame #9: torch::autograd::VariableType::gru(at::Tensor const&, at::Tensor const&, c10::ArrayRef<at::Tensor>, bool, long, double, bool, bool, bool) const + 0x4a9 (0x7fb65d567159 in /home/pinouchon/pycharms_remote/valx_trader/.venv/lib/python3.6/site-packages/torch/lib/libtorch.so.1)
frame #10: <unknown function> + 0x25fb0a (0x7fb69d960b0a in /home/pinouchon/pycharms_remote/valx_trader/.venv/lib/python3.6/site-packages/torch/lib/libtorch_python.so)
frame #11: /home/pinouchon/pycharms_remote/valx_trader/.venv/bin/python3() [0x5030d5]
frame #12: _PyEval_EvalFrameDefault + 0x449 (0x506859 in /home/pinouchon/pycharms_remote/valx_trader/.venv/bin/python3)
frame #13: /home/pinouchon/pycharms_remote/valx_trader/.venv/bin/python3() [0x504c28]

[lots more frames]

frame #60: /home/pinouchon/pycharms_remote/valx_trader/.venv/bin/python3() [0x502f3d]
frame #61: _PyEval_EvalFrameDefault + 0x449 (0x506859 in /home/pinouchon/pycharms_remote/valx_trader/.venv/bin/python3)
frame #62: /home/pinouchon/pycharms_remote/valx_trader/.venv/bin/python3() [0x504c28]
frame #63: _PyFunction_FastCallDict + 0x2de (0x501b2e in /home/pinouchon/pycharms_remote/valx_trader/.venv/bin/python3)

Any ideas?