Is your training running fine on a single GPU?
If the loss is consuming so much memory, I would assume a single GPU should consume the same amount.



 che85:

DataParallel has to have a mechanism to predict how much memory will be used by a model as well as the loss(criterion), right?


Not really, as the computation graph is defined dynamically based on your forward pass.