Hi all, @patrickvonplaten,
I’m using Wav2Vec2ForCTC.from_pretrained(“facebook/wav2vec2-large-xlsr-53”) to fine-tune my own spanish dataset (I’m actually using it in other ASR framework). I’ve limited the dataset to 100 hours of records, 105.150 utterances in the range of 1 to 10 seconds.
I’m following the example from this notebook:  Fine-tuning XLS-R for Multi-Lingual ASR with  Transformers  by @patrickvonplaten.
My issue is that that the training loss and validation loss steadily decrease first few steps and then all metrics start to worsen:

image777×887 94.9 KB

My paremeters and config is:
from transformers import Wav2Vec2ForCTC
model = Wav2Vec2ForCTC.from_pretrained(
    "facebook/wav2vec2-xls-r-300m", 
    layerdrop=0.05,
    mask_time_prob=0.0075,
    ctc_loss_reduction="mean", 
    pad_token_id=processor.tokenizer.pad_token_id,
    mask_time_length=4,
    vocab_size=len(processor.tokenizer)
)

training_args = TrainingArguments(
  output_dir=repo_name,
  group_by_length=True,
  per_device_train_batch_size=16,
  gradient_accumulation_steps=2,
  evaluation_strategy="steps",
  num_train_epochs=100,
  gradient_checkpointing=True,
  fp16=True,
  save_steps=1000,
  eval_steps=500,
  logging_steps=50,
  learning_rate=3e-4,
  warmup_steps=4000,
  save_total_limit=2,
  push_to_hub=False,
)