I wouldn’t try to use seeds in order to initialize both models etc., as the implementation of the random number generator might be different in the worst case and you would waste a lot of time.
If you want to compare both models, I would start with the good performing model (i.e. the TF model in your case), create a random input and compute the output.
Once this is done, store all data in as numpy arrays (i.e. the input, output, model parameters and buffers).
Afterwards create the corresponding PyTorch model and load the parameters and buffers manually via:
with torch.no_grad():
    model.layer.parameter.copy_(torch.from_numpy(param))
    ...

perform the forward pass next and compare the outputs of the TF model and your current PyTorch implementation.
If they show a large difference, you could try to narrow down the layer, which is creating this error.