OK, sorry about the delay, but here is a traceback showing the memory failure:
THCudaCheck FAIL file=/pytorch/aten/src/THC/generic/THCStorage.cu line=58 error=2 : out of memory
Traceback (most recent call last):
File “semantic_matching.py”, line 849, in 
train_model(model, dataset)
File “semantic_matching.py”, line 766, in train_model
pos_energy = parallel_model(*pos_batch)
File “/iscsi/rdata/rbeaudoin/projects/sme-memory/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 491, in call
result = self.forward(*input, **kwargs)
File “/iscsi/rdata/rbeaudoin/projects/sme-memory/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py”, line 113, in forward
replicas = self.replicate(self.module, self.device_ids[:len(inputs)])
File “/iscsi/rdata/rbeaudoin/projects/sme-memory/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py”, line 118, in replicate
return replicate(module, device_ids)
File “/iscsi/rdata/rbeaudoin/projects/sme-memory/lib/python3.6/site-packages/torch/nn/parallel/replicate.py”, line 12, in replicate
param_copies = Broadcast.apply(devices, *params)
File “/iscsi/rdata/rbeaudoin/projects/sme-memory/lib/python3.6/site-packages/torch/nn/parallel/_functions.py”, line 17, in forward
outputs = comm.broadcast_coalesced(inputs, ctx.target_gpus)
File “/iscsi/rdata/rbeaudoin/projects/sme-memory/lib/python3.6/site-packages/torch/cuda/comm.py”, line 40, in broadcast_coalesced
return torch._C._broadcast_coalesced(tensors, devices, buffer_size)
RuntimeError: cuda runtime error (2) : out of memory at /pytorch/aten/src/THC/generic/THCStorage.cu:58
I hope this helps.