Hi all,
I am training an image recognition model with dataset size (4M training images 200x200 size)
Here are the configurations of the training setup:

pytorch v0.4.1
multi-GPU - 4
num_workers of my dataloader = 16
tried pin_memory=true / pin_memory=false
system configuration: 4 Tesla GPUs (6GB each) RAM: 128GB

My training crashes after a few epochs with the error messages like:
ERROR: Unexpected bus error encountered in worker. This might be caused by insufficient shared memory (shm)
Here are my system’s shared memory limits:
$ipcs -lm
------ Shared Memory Limits --------
max number of segments = 8192
max seg size (kbytes) = 18014398509465599
max total shared memory (kbytes) = 18446744073642442748
min seg size (bytes) = 1

Any ideas why I am getting shared memory crashes?
Any help is highly appreciated.
Thanks