Fix CTRL tests
# What does this PR do?

Fix ctrl tests failed due to GPU memory issue. The fix is the same as in #16881



Job page (failed test)
https://github.com/huggingface/transformers/runs/6682129553?check_suite_focus=true
I verified the fix on GCP VM.<|||||>_The documentation is not available anymore as the PR was closed or merged._