It’s hard to tell any differences between your PyTorch and TF code without any code samples.
Did you implement both solutions on your own or did you use some from e.g. github?
In the latter case did you compare the network structure, i.e. number of kernels, kernel size, etc.?