I would recommend to create a single conv layer (or any other layer with parameters) in both frameworks, load the weights from TF to PyTorch, and verify that the results are equal for the same input. Once this works, you could then test blocks until you narrow down where the difference in results is caused.