That’s not really easy to answer, since the bottleneck might come from different parts.
E.g.

the data loading pipeline migth be slower in PyTorch, if you don’t use multiple workers. I don’t know what kind of data loading approach TF is using.
Are you using the CPU or GPU? In the latter case, are you using cudnn in both frameworks and torch.backends.cudnn.benchmark = True in PyTorch? Also which CUDA and cudnn version and GPUs are you using?