I remember once I transferred the weights from TF to PyTorch for a project that I first did in TF, and then switched to PyTorch. One thing that you need to pay attention to, is that convolution is implemented slightly differently in TF and PyTorch. Therefore, you have to do rotate the weights or something like that.
First, try to establish what transformation needs to be applied to the convolution weights to get the same results across the two. For this, it’s easier to do that with a small convolution layer. Then, you can transform the entire model weights.