Hi,
I’m using Wav2Vec2ForCTC.from_pretrained(“facebook/wav2vec2-base”) to fine-tune on a English language medical translation dataset, which is about 6GB. It is similar to the Timit_ASR dataset, with the exception that the wav files are in 48KHz. I’m following the example show in this notebook: Fine-Tune Wav2Vec2 for English ASR in Hugging Face with 🤗 Transformers 1
Thank you @patrickvonplaten for an excellent illustration.
My issue is that that the training loss and validation loss show either nan or inf, and the WER does not decrease.
/usr/local/lib/python3.7/dist-packages/torch/optim/lr_scheduler.py:134: UserWarning: Detected call of `lr_scheduler.step()` before `optimizer.step()`. In PyTorch 1.1.0 and later, you should call them in the opposite order: `optimizer.step()` before `lr_scheduler.step()`.  Failure to do this will result in PyTorch skipping the first value of the learning rate schedule. See more details at https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate
  "https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate", UserWarning)

 [5895/5895 24:03, Epoch 1/1]
Step	Training Loss	Validation Loss	Wer	Runtime	Samples Per Second
500	4.942800	nan	1.000000	20.818400	18.301000
1000	3.023500	nan	1.000000	20.785500	18.330000
1500	inf	3.477225	1.000000	20.757200	18.355000
2000	nan	3.492772	1.000000	20.922500	18.210000
2500	nan	nan	1.000000	20.817700	18.302000
3000	nan	nan	1.000000	20.788900	18.327000
3500	nan	nan	1.000000	20.850800	18.273000
4000	nan	nan	1.000000	20.928300	18.205000
4500	nan	3.214458	1.000000	20.900100	18.230000
5000	nan	nan	1.000000	20.907300	18.223000
5500	nan	3.297620	1.000000	20.979900	18.160000
TrainOutput(global_step=5895, training_loss=nan, metrics={'train_runtime': 1444.0059, 'train_samples_per_second': 4.082, 'total_flos': 2.494079663135328e+17, 'epoch': 1.0, 'init_mem_cpu_alloc_delta': 1760702464, 'init_mem_gpu_alloc_delta': 377847808, 'init_mem_cpu_peaked_delta': 0, 'init_mem_gpu_peaked_delta': 0, 'train_mem_cpu_alloc_delta': 2907852800, 'train_mem_gpu_alloc_delta': 1116327936, 'train_mem_cpu_peaked_delta': 847519744, 'train_mem_gpu_peaked_delta': 2587473920})

The only exceptions that I have to the notebook code is that I’m using torchaudio instead of soundfile and I’m downsampling to 16 KHz. The down sampling seems to be working correctly because the audio plays well when checking. That code is here:
import torchaudio

def speech_file_to_array_fn(batch):
    speech_array, sampling_rate = torchaudio.load(batch["file_name"])
    batch["speech"] = speech_array[0].numpy()
    batch["sampling_rate"] = sampling_rate
    batch["target_text"] = batch["phrase"]
    return batch

dataset = dataset.map(speech_file_to_array_fn, remove_columns=dataset.column_names["train"], writer_batch_size=32, num_proc=4)

import librosa
import numpy as np

def resample(batch):
    batch["speech"] = librosa.resample(np.asarray(batch["speech"]), 48_000, 16_000)
    batch["sampling_rate"] = 16_000
    return batch

dataset = dataset.map(resample, writer_batch_size=32, num_proc=4)

I’ve tried to use different learning rates. A couple of the 500 increment steps in the above table actually showed a loss number instead of nan. But then subsequent losses were nan.
I also tried to follow the example in the Fine-tuning XLSR-Wav2Vec2 for Multi-Lingual ASR with  Transformers notebook. That resulted in a different error saying that the blank was out of label range.
I think I have a vanishing/exploding gradient problem. Perhaps I should try changing learning rates and regularization like dropout. I’m not sure how to make it work in this notebook.
Any help would be appreciated.
The dataset is from here: Medical Speech, Transcription, and Intent | Kaggle
Notebook is run on Google Colab Pro.
Any help would be much appeciated.
Thanks,
Sidd