I don’t know how you’ve loaded the parameters from TF to PyTorch, but would recommend to check the layers in isolation and make sure e.g. the memory layout, flipping etc. are right.
In your PyTorch model you re using nn.Softmax as the last activation, which is usually wrong. Guessing you are working on a multi-class classification using nn.CrossEntropyLoss, remove the softmax as the criterion expect raw logits.