Hi,
I train a cnn network with batch normalization layers where my loss function is based on L2 distance.
When I use DataParallel to train the network,  the evaluation results of the network dramatically fail in eval() mode compared to train() mode.
I understand the difference between eval() and train() and that their performance may differ at a certain level; however, I still think that this issue is related to DataParallel since when I train the model without DataParallel, i.e., with a single gpu, the evaluation results in eval() and train() modes remain almost identical.