Yes, transferring the weights from Keras to PyTorch does decrease the AUROC (which is my metric) 0.05 which is big as far as I am concerned.
And yes, I have set the model to eval before evaluating.
layers consist of Relu, conv2D, linear.