You could check the default parameter initialization in both frameworks and make sure your ported PyTorch model uses the same as your TF model.
If the model is still diverging, you could verify the architecture by loading the TF parameters into the PyTorch model (or vice versa) and check the output for a fixed input tensor.