Blockquote
so why 0.001 lr works well in keras

It could be any number of factors. You’d have to investigate the source code for every layer to see if it is implemented exactly the same as in PyTorch. Maybe something simple like a learning rate decay is implemented by default in Keras.

Blockquote
what is your advice for my to do?

Try training a pytorch model with a lower learning rate, until you actually observe a consistent decrease in training loss throughout training. Don’t worry about valid accuracy for now. If train loss does not decrease with a lower LR, then something is wrong with either the data, the labels, the model architecture or the training script. I can’t really help much for debugging these. Just remember that PyTorch will not necessarily do everything in exactly the same way as Keras by default.