Fine-tuning Wav2Vec2: Concern that pretrained weights are being reinitialized
## Environment info

- OS Type: Google Colab
- 'transformers' version: 4.14.1
- Platform: Linux-5.4.104+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.12
- PyTorch version (GPU?): 1.10.0+cu111 (True)
- Tensorflow version (GPU?): 2.7.0 (True)

### Who can help
Anyone with knowledge of fine-tuning Wav2Vec2 but specifically @patrickvonplaten, @anton-l 

## Information

Model I am using: wav2vec2-large-960h-lv60

The tasks I am working on is:
* My own task or dataset: fine-tuning wav2vec2 on technical jargon

Hi! Hope this message finds the reader well. I've been attempting to fine-tune Wav2Vec2 on a new domain where highly technical jargon is used in a noisy environment. Out-of-the-box with a language model I'm getting fairly good WER scores but my goal is to help the model be more robust to background noise interference.

The problem arises when using:
* The official example scripts
My code is roughly based off of Patrick von Platen's [Fine-Tune Wav2Vec2 for English ASR with Huggingface Transformers](https://huggingface.co/blog/fine-tune-wav2vec2-english) blog post. It's been slightly modified to handle my own data format but the training pipeline has not been altered.

## Description of Behavior
Our data is still coming in so we don't have a lot to work with (<7 min so far) but when I fine-tune on what little we do have to evaluate initial performance, it appears like the model is starting from scratch (no pre-training) as it no longer predicts words but random letter sequences. I was concerned that maybe my learning rate (or noisy environment) was the cause of the issue so I did a series of small experiments with a clean librispeech subsample of ten audio files (train size = 10, test size = 10), lowering the learning rate for each experiment but all the results were identical to the following:

Average audio file duration: 13.34 seconds
Average WER: 100%
TARGET: IN HIS MORE SOBER MOMENTS HE WAS NOT ALWAYS ABLE TO ASSUME THAT APPEARANCE OF EQUALITY WITH HIS COMPANIONS WHICH IT WAS THE AMBITION OF HIS SOUL TO ACHIEVE BUT A SECOND GLASS OF WHISKY AND WATER
PREDICTION: TOETITSTOTZTITBTHT'TKTITZTHTATKT'TITBTHTBTKETMTZITSTKTITJTNTZITETHTMTITNTYTJTNGTZTITNTATYTKITMTHTITNZTZTLTBKTITMSTNTMITNXTXTKTNT'TNETPTKITHTRTITKTTLTNTYTOTMTGTITJTOMSITSTOTZTITPTHBTXTNTETOTHTETZITJSTOTPSITOMTITJTNTZTITMSTKTITNBTATOTMTOTHETITHTRITSTOTZTITZTHLTYTITMTHTITNTPTSTOTKTCTKITATLTMITNTITZTKTPTHETFTIT TYTNTZTZTITHRTITJSTOTZTUTGTITNETFITJTNTMTK'TIT

I then thought maybe something was being re-intialized that should not be. I traced where the linear head was initialized and where the pre-trained weights were loaded in the Huggingface Transformers repository but saw nothing out of place. I also followed the learning rate value all the way to training then during training to see if the learning rate changes were not being kept and that was why changing the learning rate had no effect. Again, nothing looked incorrect and the new learning rate was stored consistently across the different scripts.

I then wondered what would happen if more data was added since others (for example [this one](https://www.tensorflow.org/hub/tutorials/wav2vec2_saved_model_finetuning)) were saying smaller datasets were experiencing these kind of random predictions. So, I incrementally added more librispeech samples and increased the number of epochs until I reached roughly 15-20 minutes of data and 30 epochs at which point the model started overfitting:

TARGET: BUT STOPPED SHORT STUPEFIED AND FRIGHTENED WITHIN THREE STEPS OF HATTERAS WHO STARTED UP THAT MOMENT AND THROWING OFF HIS DISGUISE KNELT ON ONE KNEE AND AIMED STRAIGHT AT THE BEAR'S HEART

PREDICTION: BUT STOPPED SHORT STUPEFIED AND FRIGHTENED WITHIN THREE STEPS OF HATTERAS WHO STARTED UP AT THAT MOMENT AND THROWING OFF HIS DISGUISE KNELT ON ONE KNEE AND AIMED STRAIGHT AT THE BEAR'S HEART

Based on this it seems that more gradient steps (roughly 30000) plus a lot more data is needed in order to have any good results...but this seems odd to me. I understand that, with so little data, it shouldn't learn much about the new domain yet but shouldn't it also not lose all the pre-trained knowledge within a few gradient steps and with a conservative learning rate?


## Questions:
1. Is it purely my own misunderstanding on how fine-tuning works for Wav2Vec2? My assumption was that the training would start from the pre-trained checkpoint and therefore should already have some knowledge about speech and English from the start of fine-tuning. Are the predictions initially random due to the new LM head established in the Wav2Vec2ForCTC class (in other words is this normal)? If this is abnormal then what can I do to resolve this issue?

2. Why does altering the learning rate not have any impact if this is a normal behavior?

## To reproduce
My Google Colab notebook and the sample data used (audio, CSVs) are included here in a zip file. All you need to do is:

1. Unzip it in your Google drive at the directory /content/drive/MyDrive so you have the correct path /content/drive/MyDrive/demo_data with the audio files and CSVs within it.
2. Run every cell in the Colab Notebook
3. The last cell will show you predictions from the model


This might be an obvious question but I've been struggling with it for a while so I really appreciate any insight or clarification anyone can give me on it! Thanks in advance.
[demo_data-coulter.zip](https://github.com/huggingface/transformers/files/7757475/demo_data-coulter.zip)


Hey @rcoulter13,

This issue is quite difficult to debug for us since it's not an obvious bug but deals with the model having problems with continual learning. It's quite a common phenomena that fine-tuned models have catastrophic forgetting when they are re-finetuned on out of distribution data<|||||>There are a ton of reasons why the fine-tuning might not work. Some include:
- the target labels (text) includes many out-of-vocabulary tokens. The wav2vec2 Librispeech models are all fine-tuned on very clean data that has all punctuation removed and usually on capitalized letters. You should check that this is also the case for your data
- Maybe freezing the whole base model can help mean ing to set `require_grad` to False for all layers except the `lm_head`
- Other hyper-parameters for training
- One should also make sure that the input data is correctly sampled at 16kHz
<|||||>Also note that the official blog post is based on TIMIT which is a very small and clean dataset<|||||>Hi @patrickvonplaten,

Thank you for addressing my issue so quickly and for your detailed suggestions! I appreciate your time.

1. My target labels indeed match the original target labels, all punctuation has been removed and capitalization is all uppercase - I had the same thought so I have already double-checked this.
2. I have an assertion statement that checks that all the input data is 16kHz and I've also manually checked the data with SoX
3. I hadn't thought of freezing the entire model except the lm_head but that's an excellent idea - thank you!

So, just to make sure I understand, you're saying the behavior in my model isn't standard? Part of me was wondering if it simply just requires that many epochs for the new head to learn anything and then for it to update the rest of the model. But you're saying we should be seeing much better results much earlier in training with the pre-trained weights then?<|||||>To be honest, I've never continued fine-tuned for a speech model yet, so I really don't know what to expect here. I usually throw away the `lm_head`, keep the base model, randomly initialize the `lm_head` and fine-tune the new model on the new data. But it might make sense to continue fine-tuning in your case here.

Gently pinging some other people here in case they have continued fine-tuning for Wav2Vec2 and might have an idea @anton-l @flozi00 @jonatasgrosman<|||||>Good to know, I'll definitely try your suggestion of freezing the entire model minus the lm_head and see where that takes me. Thanks again for your help and happy holidays!<|||||>Just wanted to followup for anyone else who might run across this. I ended up not having to freeze the whole base model. My problem was within my processor. For my task I only needed to use the vocab Wav2Vec2 was trained on (I'm not adding any new tokens) so I saw the immediate benefit of the pre-trained model on the first epoch of fine-tuning when I made sure the vocabulary and its corresponding numeric value were identical to Wav2Vec2's pre-trained vocab (i.e. ```vocab = {"[PAD]" : 0, "<s>" : 1 ... "Z" : 31}```). One easy way to check if yours matches is to save the pre-trained vocab to a local directory and view the vocab.json file. So if the vocab is identical and the data is clean (quiet environment) one should see the benefit of the whole base model on epoch 1 and improvement from fine-tuning within 10-20 epochs. If one isn't seeing this behavior then there might be a problem with the labels or the training data similar to what I experienced. Of course, it might be something else entirely but I'd check these two scenarios first. Hope this is helpful.