Fix `Wav2Vec2` CI OOM 
# What does this PR do?

After #23813, we get OOM for `tests/models/wav2vec2/test_modeling_wav2vec2.py::Wav2Vec2ModelIntegrationTest::test_wav2vec2_with_lm_invalid_pool` (when running the whole `wav2vec2` test suite)

Just doing some cleaning up and no more OOM.
_The documentation is not available anymore as the PR was closed or merged._