The first row is my print command

Initiating epoch 0
5f8f287430a2:27239:27239 [0] NCCL INFO Bootstrap : Using [0]eth0:172.17.0.2<0>
5f8f287430a2:27239:27239 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
5f8f287430a2:27239:27239 [0] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]
5f8f287430a2:27239:27239 [0] NCCL INFO NET/Socket : Using [0]eth0:172.17.0.2<0>
5f8f287430a2:27239:27239 [0] NCCL INFO Using network Socket
NCCL version 2.7.8+cuda10.2
5f8f287430a2:27239:27499 [0] NCCL INFO Channel 00/02 :    0   1   2
5f8f287430a2:27239:27500 [1] NCCL INFO threadThresholds 8/8/64 | 24/8/64 | 8/8/64
5f8f287430a2:27239:27501 [2] NCCL INFO threadThresholds 8/8/64 | 24/8/64 | 8/8/64
5f8f287430a2:27239:27499 [0] NCCL INFO Channel 01/02 :    0   1   2
5f8f287430a2:27239:27500 [1] NCCL INFO Trees [0] 2/-1/-1->1->0|0->1->2/-1/-1 [1] 2/-1/-1->1->0|0->1->2/-1/-1
5f8f287430a2:27239:27501 [2] NCCL INFO Trees [0] -1/-1/-1->2->1|1->2->-1/-1/-1 [1] -1/-1/-1->2->1|1->2->-1/-1/-1
5f8f287430a2:27239:27500 [1] NCCL INFO Setting affinity for GPU 1 to ffff,0000ffff
5f8f287430a2:27239:27501 [2] NCCL INFO Setting affinity for GPU 3 to ffff0000,ffff0000
5f8f287430a2:27239:27499 [0] NCCL INFO threadThresholds 8/8/64 | 24/8/64 | 8/8/64
5f8f287430a2:27239:27499 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1|-1->0->1/-1/-1 [1] 1/-1/-1->0->-1|-1->0->1/-1/-1
5f8f287430a2:27239:27499 [0] NCCL INFO Setting affinity for GPU 0 to ffff,0000ffff
5f8f287430a2:27239:27500 [1] NCCL INFO Could not enable P2P between dev 1(=5e000) and dev 0(=3b000)
5f8f287430a2:27239:27499 [0] include/shm.h:28 NCCL WARN Call to posix_fallocate failed : No space left on device
5f8f287430a2:27239:27500 [1] include/shm.h:28 NCCL WARN Call to posix_fallocate failed : No space left on device
5f8f287430a2:27239:27501 [2] include/shm.h:28 NCCL WARN Call to posix_fallocate failed : No space left on device
5f8f287430a2:27239:27499 [0] NCCL INFO include/shm.h:41 -> 2
5f8f287430a2:27239:27500 [1] NCCL INFO include/shm.h:41 -> 2
5f8f287430a2:27239:27501 [2] NCCL INFO include/shm.h:41 -> 2
5f8f287430a2:27239:27499 [0] include/shm.h:48 NCCL WARN Error while creating shared memory segment nccl-shm-recv-ea821c4e67e70b28-0-2-0 (size 9637888)
5f8f287430a2:27239:27500 [1] include/shm.h:48 NCCL WARN Error while creating shared memory segment nccl-shm-recv-ea821c4e67e70b28-0-0-1 (size 9637888)
5f8f287430a2:27239:27499 [0] NCCL INFO transport/shm.cc:101 -> 2
5f8f287430a2:27239:27500 [1] NCCL INFO transport/shm.cc:101 -> 2
5f8f287430a2:27239:27501 [2] include/shm.h:48 NCCL WARN Error while creating shared memory segment nccl-shm-recv-ea821c4e67e70b28-0-1-2 (size 9637888)
5f8f287430a2:27239:27499 [0] NCCL INFO transport.cc:30 -> 2
5f8f287430a2:27239:27501 [2] NCCL INFO transport/shm.cc:101 -> 2
5f8f287430a2:27239:27500 [1] NCCL INFO transport.cc:30 -> 2
5f8f287430a2:27239:27501 [2] NCCL INFO transport.cc:30 -> 2
5f8f287430a2:27239:27499 [0] NCCL INFO transport.cc:49 -> 2
5f8f287430a2:27239:27500 [1] NCCL INFO transport.cc:49 -> 2
5f8f287430a2:27239:27501 [2] NCCL INFO transport.cc:49 -> 2
5f8f287430a2:27239:27499 [0] NCCL INFO init.cc:766 -> 2
5f8f287430a2:27239:27501 [2] NCCL INFO init.cc:766 -> 2
5f8f287430a2:27239:27500 [1] NCCL INFO init.cc:766 -> 2
5f8f287430a2:27239:27501 [2] NCCL INFO init.cc:840 -> 2
5f8f287430a2:27239:27500 [1] NCCL INFO init.cc:840 -> 2
5f8f287430a2:27239:27499 [0] NCCL INFO init.cc:840 -> 2
5f8f287430a2:27239:27501 [2] NCCL INFO group.cc:73 -> 2 [Async thread]
5f8f287430a2:27239:27500 [1] NCCL INFO group.cc:73 -> 2 [Async thread]
5f8f287430a2:27239:27499 [0] NCCL INFO group.cc:73 -> 2 [Async thread]
5f8f287430a2:27239:27239 [0] NCCL INFO init.cc:906 -> 2
Traceback (most recent call last):
File “main_comb.py”, line 75, in 
[trloss,tstloss,tstevm,inf_time]=cnn.run_epoch(epoch,show=True)
File “/common_space_docker/storage_4TSSD/ben/ce/Results/Nov_19_20_11:02_3DFF_base/cnn_imp.py”, line 875, in run_epoch
train_loss = train_model()
File “/common_space_docker/storage_4TSSD/ben/ce/Results/Nov_19_20_11:02_3DFF_base/cnn_imp.py”, line 762, in train_model
outputs = model(channel)
File “/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py”, line 727, in _call_impl
result = self.forward(*input, **kwargs)
File “/usr/local/lib/python3.6/dist-packages/torch/nn/parallel/data_parallel.py”, line 160, in forward
replicas = self.replicate(self.module, self.device_ids[:len(inputs)])
File “/usr/local/lib/python3.6/dist-packages/torch/nn/parallel/data_parallel.py”, line 165, in replicate
return replicate(module, device_ids, not torch.is_grad_enabled())
File “/usr/local/lib/python3.6/dist-packages/torch/nn/parallel/replicate.py”, line 88, in replicate
param_copies = _broadcast_coalesced_reshape(params, devices, detach)
File “/usr/local/lib/python3.6/dist-packages/torch/nn/parallel/replicate.py”, line 71, in _broadcast_coalesced_reshape
tensor_copies = Broadcast.apply(devices, *tensors)
File “/usr/local/lib/python3.6/dist-packages/torch/nn/parallel/_functions.py”, line 22, in forward
outputs = comm.broadcast_coalesced(inputs, ctx.target_gpus)
File “/usr/local/lib/python3.6/dist-packages/torch/nn/parallel/comm.py”, line 56, in broadcast_coalesced
return torch._C._broadcast_coalesced(tensors, devices, buffer_size)
RuntimeError: NCCL Error 2: unhandled system error

These are 3 TITAN GPUs with more than enough space (a batch and the model is about 4Gb of 25Gb available on each TITAN)