kaladin:

Currently, I am thinking that it has something to do with how the weights for the various layers are initialized


If you are starting from randomly initialized weights, then it is completely natural to have different outputs.
If you load a pre-trained model, then the weights for both models (TF and PyTorch) should be the same, so in that case, you should not get different outputs.
Another thing to consider, it might be that the two pre-trained models need certain input normalization, for example, the range of pixel values be within [0,1] or [-1,1] instead of [0, 255]. Or possibly, they would also need centering the color channels in the input using certain mean values. So, if you are using a pre-trained model, you would need to figure out what normalization steps need to be applied to the input images.