@lysandre has a far better solution to this issue. See this Github post can't allocate memory error with wav2vec2 · Issue #10366 · huggingface/transformers · GitHub 18
Code screen grab below 
lysandre1617×908 94.1 KB