removing torch.cuda.empty_cache() from TF function
torch.cuda.empty_cache() was being called from a TF function (even when torch is unavailable)
not sure any replacement is needed if TF OOMs

simply running the benchmarks on a GPU with lower HBM will reproduce this error
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3267?src=pr&el=h1) Report
> Merging [#3267](https://codecov.io/gh/huggingface/transformers/pull/3267?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/cc4c37952a961f2d13e83f3d5ba6dab811d0bbfd&el=desc) will **not change** coverage by `%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/3267/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/3267?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #3267   +/-   ##
=======================================
  Coverage   77.82%   77.82%           
=======================================
  Files          98       98           
  Lines       16666    16666           
=======================================
  Hits        12970    12970           
  Misses       3696     3696           
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/3267?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3267?src=pr&el=footer). Last update [cc4c379...4bb2bc3](https://codecov.io/gh/huggingface/transformers/pull/3267?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
<|||||>Indeed, thanks!