There was an issue with the shape of my data. It works fine after fixing the bug! But to add some info for someone else who might be having a similar problem, cleaning the speech before finetuning could help!
  
    
    
    Wav2vec2 not converging when finetuning 🤗Transformers
  
  
    Oh I see. 
I think wav2vec is having a hard time because the audio is so different to the one it was pretrained on.