Hi there everyone 
I’m currently trying to train a Wav2Vec base model. During the pre-training phase, The loss starts off around 4, decreases and then shoots up to 6.658 and stays there. The accuracy is also low and does not increase. My learning rate is set at 0.005. I started off with a learning rate of 0.0001 and started increasing it gradually when I saw these results. I use the english Wav2Vec model for weight initialisation. I thought it would improve if I waited longer but it stays the same even after 20 epochs. Can anyone please share some advice on what I could do to avoid this and improve the training?
Your assistance will be much appreciated!