I would recommend to start with a simple code base and compare the Keras results to your PyTorch model.
E.g. isolate the model and use some dummy data to compare the model outputs.
Your current code is pretty long, which makes debugging hard.