[Do not merge] Chase more pt tf inconsistency
# What does this PR do?


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_15321). All of your documentation changes will be reflected on that endpoint.<|||||>In terms of running time

- Circle CI
  - current : [56.11s](https://circleci.com/api/v1.1/project/github/huggingface/transformers/384755/output/112/0?file=true&allocation-id=62238c1d4f148c1110a1efa8-0-build%2F1610CB69)
  - this PR : [61.21s](https://circleci.com/api/v1.1/project/github/huggingface/transformers/384731/output/112/0?file=true&allocation-id=622385347ccc3f3894ec7697-0-build%2F351FF724)