Dear Madam/Sir,
Recently, I am training a model with nn.Embedding, nn.LSTM, nn.Linear modules.
DataParallel is also leveraged for training efficiency. However, when multiple GPUs are utilized (with the same batch size as the single GPU running), the model is hard to train, i.e. the loss decreases very slowly, and the final model cannot work.
For optimizer, we directly use the codes like in imageNet example:

        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

Thank you for helping me.
Best,
Yikang