There is no way for each worker to have its unique set of data, so that each one does not need  a full copy of the original ?
I mean splitting the dataset between every worker ?
Also what I the reason the validation mode is way more efficient on CPU memory than training, because on CPU side, this is basically the same to me, I mean the data processing is the same, but it uses 10 GB in RAM.
only differences are :
# instead of model.train()
model.eval()

# using volatile
input_25 = torch.autograd.Variable(input[0], volatile=True).cuda()