I’ve been playing around with the XLSR-53 fine-tuning functionality but I keep getting nan training loss.
Audio files I’m using are:

Down-sampled to 16kHz
Set to one channel only
Vary in length between 4 to 10s

I’ve set the following hyper-params:

attention_dropout=0.1
hidden_dropout=0.1
feat_proj_dropout=0.0
mask_time_prob=0.05
layerdrop=0.1
learning rate:

on a warmup schedule to 3e-4 for 3 epochs
at 5e-4 for 3 epochs
back to 3e-4




Sadly, I’m fine-tuning the model on an unpublished corpus, so I am probably not at liberty to upload it here which might hinder reproducibility efforts greatly.
Here’s what the loss and WER progression looks like:

image497×815 75.2 KB

Anyone know what could be happening here? The model seems to be training just fine and some testing proves that the model performs well on the language I’m training it on. So what’s up with the training loss?
Pinging @patrickvonplaten and @valhalla as this might be relevant to them.