The trick is that if you compute the loss outside DataParallel, outputs are collected in a single GPU. Then loss is calculated and backpropagated.
If you compute the loss inside DataParallel, instead of returning huge amount of data (that will be collected in a single gpu), you just return few floats.
Loss is unsqueezed because dataparallel works with batches, so it fails at the time of returning a non-dimensional tensor. Besides, non-dimensional tensors are supported in last versions of PyTorch, but its not something implemented in the early design.
You can set your model, optimizer parameters etcetera and then call the function. As you can see it requires a model as input so it’s agnostic.
At the time of saving I would save the base model.
If you check the state dict it will appear couple more roots, like datap.module.fullmodel.model
so you can save datap.module.fullmodel.model.state_dict()
Check that deeper