Remove attention from padding
# What does this PR do?

This PR removes all attention from padding tokens where before there was a minimal ammount. This promotes reproducibility since it doesn't matter the batch size but one should get the same results every time from any task (which now is not the case).

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@LysandreJik

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_15866). All of your documentation changes will be reflected on that endpoint.<|||||>Thank you for your PR! See issue here: https://github.com/huggingface/transformers/issues/14859
Ideally we would want all models to behave similarly, I believe @ydshieh wanted to work on it.<|||||>Thank you @nicola-decao for this PR!

This large negative values for attention mask is one thing in our next steps to clean up. In particular:

  - we will need the same value for any specific model across frameworks (PyTorch, TensorFlow, and Flax)
  - we also prefer to use the same value among all models

Also, using `-float("inf")` is theoretically good, but it seems there are some disadvantages in practice. (I need to investigate this part a bit more to be sure).

Because of the above considerations, I prefer not to merge this PR, and our team will work on this soon.
I would like to hear from @patrickvonplaten about this decision.

<|||||>@nicola-decao As explained in my previous comment, I am going to close this PR without merge.

Thank you again for your effort and opening this PR, very appreciated ❤️.

Don't hesitate if you have more comments.<|||||>@ydshieh @patrickvonplaten  yeah I understand! Although we should try to address this at some point since it is hard to debug or inspect models where results are not the same depending on padding. It make a tiny difference (ie `exp(-1e9)` is very small so it is ok but still people need to use `torch.isclose` instead of equality since it is not the same)<|||||>@nicola-decao Do you mean you want to have `equality` instead of using `torch.isclose`?<|||||>> @nicola-decao Do you mean you want to have `equality` instead of using `torch.isclose`?

@ydshieh  Correct! Right now if you use different padding lengths the results might differ for the same input which is not ideal for deterministic / reproducible results.