Hi,
This may happen due to difference in default values in models. Here is some differences that I can think of as I do not know how Keras works.

PyTorch DataLoader does not shuffle the data by default but in your task, it seems you can do it, so this can help a lot.
I do not know how you constructed dataset, but normalization might be needed if already isn’t.
This might not be much important but PyTorch uses uniform weight initalization for linear and conv layers. You might want to make sure this also correct for your case.

And again, I think you need to check the default value in every object you have initialized here as there might be different conventions.
Bests