You could profile different parts of the code and try to narrow down where the speed difference comes from.
E.g. if you think that TF’s prefetching might play an important role, you could completely remove the data loading pipeline and use random inputs.