OK it  must be the batch size, in Keras I do not specify the batch size it is handled dynamically, if I remove batch size in Pytorch it is a lot worse:
Train Loss: 4.564 | Accuracy: 30.105
Train Loss: 4.564 | Accuracy: 30.105
Train Loss: 4.564 | Accuracy: 30.105
Train Loss: 4.564 | Accuracy: 30.105