Hmm I’m having this issue all of a sudden where my training is suddenly throwing these errors:
Using cuda
  1%|▏         | 21/1584 [00:00<01:00, 26.03it/s]
Process finished with exit code 139 (interrupted by signal 11: SIGSEGV)

It seems like it gets through some iterations and then bam, segmentation fault.
I checked nvidia-smi:
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 450.119.03   Driver Version: 450.119.03   CUDA Version: 11.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  GeForce GTX 108...  Off  | 00000000:01:00.0  On |                  N/A |
|  0%   50C    P5    21W / 250W |   4867MiB / 11177MiB |      1%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

Having just upgraded to torch 1.9.0, is it possible my gpu drivers need to be updated?