ops, I just found it’s an issue about learning rate.
When I set learning rate into 4e-4, four times of the rate using on single GPU.
Epoch [1/2], Step [100/150], Loss: 1.7276
Epoch [2/2], Step [100/150], Loss: 1.2062
Training complete in: 0:00:18.275619
Dev loss: 1.1129298210144043