Avoid `GPT-2` daily CI job OOM (in TF tests)
# What does this PR do?

Clear (as much as possible) GPU memory usage allocated by torch, so the TF tests (GPT-2) get more room and make @Rocketknight1 's life easier 😆.

Some (TF) tests get OOM after #23234.

Note, the changes in this PR are in torch test files instead of TF test files! This is similar to #16881, which has more details mentioned.

Running manually and all gpt2 tests pass now.

_The documentation is not available anymore as the PR was closed or merged._