I did not transfer the backward to loss.mean().backward, with loss.mean().backward I could still train my model with loss decrease smoothly. However, the overall training and testing accuracy are slightly worse than the same model which trained without dataparallel. Any clues?
is the switch to loss.mean().backward a must?