TF-OPT attention mask fixes
With apologies for the delay, this PR should hopefully resolve the issues in #24637. @abb128 can you please try installing from this PR and verify if it resolves your issues? You can install from this PR with:

`pip install --upgrade git+https://github.com/huggingface/transformers.git@tf_opt_fixes`

Fixes #24637
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_25238). All of your documentation changes will be reflected on that endpoint.