It was a long time ago and I’ve suspended the project for now.
Below are the settings for my last experiment.
You could also look into GitHub - Edresson/Wav2Vec-Wrapper: An easy way to fine-tune Wav2Vec 2.0 for low-resource languages..
It was very helpful for me.
model = Wav2Vec2ForCTC.from_pretrained(
    "facebook/wav2vec2-large-xlsr-53",
    gradient_checkpointing=True, 
    ctc_loss_reduction="mean", 
    pad_token_id=processor.tokenizer.pad_token_id,
    vocab_size=len(processor.tokenizer),
)

model.freeze_feature_extractor()

training_args = TrainingArguments(
    output_dir="/workspace/notebooks/models/wav2vec-lt",
    group_by_length=False,
    per_device_train_batch_size=24,
    gradient_accumulation_steps=2,
    evaluation_strategy="steps",
    num_train_epochs=30,
    fp16=True,
    save_steps=1000,
    eval_steps=1000,
    logging_steps=1000,
    learning_rate=4.42184e-05,
    warmup_steps=500,
    save_total_limit=10,
    load_best_model_at_end=True,
    greater_is_better=False,
    metric_for_best_model="wer",
    weight_decay=0.0354792,
)

trainer = Trainer(
    model=model,
    data_collator=data_collator,
    args=training_args,
    compute_metrics=compute_metrics,
    train_dataset=dataset_prepared['train'],
    eval_dataset=dataset_prepared['valid'],
    tokenizer=processor.feature_extractor,
    callbacks=[EarlyStoppingCallback(early_stopping_patience=3)],
)