In case someone else gets here and has the same issue, I think that the problem is using reshape before transpose.
I have loaded TF weights with PyTorch by permuting the weight Tensor, and it worked fine.