No, the loss shouldn’t depend on the number of threads (inter-op/intra-op).
I would recommend to check the model architecture as well as the initialization of all random parameters as they differ between the frameworks.
If the loss is still diverging in PyTorch, you could load the Keras parameters into your PyTorch model and make sure to get the same output.