I got stuck with this problem forever, and finally reducing the batch size to 2 helped to improve the performance. Thank you so much!!!
I expect the model to achieve lower l2 loss though, do you have any other suggestions?