Fix slow gpu tests lysandre
Fixes three tests of the slow + gpu suite cc @sshleifer 
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4487?src=pr&el=h1) Report
> Merging [#4487](https://codecov.io/gh/huggingface/transformers/pull/4487?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/6dc52c78d8f1f96ffd9b8f8178e142b7d4a77d14&el=desc) will **increase** coverage by `0.00%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/4487/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/4487?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #4487   +/-   ##
=======================================
  Coverage   78.02%   78.03%           
=======================================
  Files         123      123           
  Lines       20477    20477           
=======================================
+ Hits        15978    15980    +2     
+ Misses       4499     4497    -2     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/4487?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4487/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `88.66% <0.00%> (+0.16%)` | :arrow_up: |
| [src/transformers/trainer.py](https://codecov.io/gh/huggingface/transformers/pull/4487/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90cmFpbmVyLnB5) | `39.76% <0.00%> (+0.23%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/4487?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4487?src=pr&el=footer). Last update [6dc52c7...2260280](https://codecov.io/gh/huggingface/transformers/pull/4487?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
