Hi all! I have just completed my initial experiment with 40 epoch and 2e-3 learning rate (the other hparams are kept the same with @patrickvonplaten demo notebook). I run the code from Kaggle since Colab Pro restricted my GPU usage due to long usage time  It tooks more than 9 hours and Kaggle also stopped working  However, validation loss and WER results did not change dramatically even though training loss decreases sharply, as seen below.

WhatsApp Image 2021-03-20 at 10.55.321600×720 148 KB

Also, keep in mind that fine tuning CNN layers does not lead to better results as suggested by the authors and demo file.  I gave it a shot but the score (50% WER in test set) was not as good as demo file, too.

WhatsApp Image 2021-03-20 at 11.02.401600×720 143 KB

Good luck with your experiments