Current state of effort:
My main model wer got stuck at 0.3656 = 3e-4 became a high learning rate. This happened in checkpoint 2600. So, I decided to use the saved weights to run two models in parallel.
First model: take original model and decrease learning rate to 0.8e-4: result so far at checkpoint 400:

wer-lr-0.08913×122 21.1 KB

Second model: comment #model.freeze_feature_extractor(). Result so far at checkpoint 400 (this will take much longer to train):

wer-unfreeze_start907×124 21 KB

Also, I had GPU issues so the training is not optimized.