[Bart: example] drop columns that are exclusively pad_token_id from input_ids
reasoning: These columns slow down computation, but do not change output.
impact: this reduces the runtime to compute EVAL on the CNN examples from 2h to 1:37 before any other changes.

I'm open to putting this as a method on `PretrainedTokenizer` if others find it useful.

@joeddav you might find this useful.


### Code for easy copy paste
```python
def trim_batch(
    input_ids, pad_token_id, attention_mask=None,
):
    """Remove columns that are populated exclusively by pad_token_id"""
    keep_column_mask = input_ids.ne(pad_token_id).any(dim=0)
    if attention_mask is None:
        return input_ids[:, keep_column_mask]
    else:
        return (input_ids[:, keep_column_mask], attention_mask[:, keep_column_mask])
```
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3400?src=pr&el=h1) Report
> Merging [#3400](https://codecov.io/gh/huggingface/transformers/pull/3400?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/f7dcf8fcea4d486544f221032625a97ad7dc5405&el=desc) will **not change** coverage by `%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/3400/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/3400?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #3400   +/-   ##
=======================================
  Coverage   77.55%   77.55%           
=======================================
  Files         100      100           
  Lines       16970    16970           
=======================================
  Hits        13161    13161           
  Misses       3809     3809           
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/3400?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3400?src=pr&el=footer). Last update [f7dcf8f...9cfd3da](https://codecov.io/gh/huggingface/transformers/pull/3400?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
