@smb thanks. But it seems the problems is not caused by the issues you mentioned.

currently the batch size is 128, i have also tried 256, the result is similar.
I set the momentum 0.01, but does not help.

I also believed this is caused by BatchNorm layer, when i drop out the model.eval() in the validation phase, the validation loss is very similar to training loss (i used the MSE loss).  However, when model.eval() is activated, the performance becomes very bad, i found sometimes the performance becomes even worse and worse as the epoch increases (quite interesting).
So could list some more potential reasons?