UPDATE:
I run the comet.ml’s adaptation of my model using native Pytorch CTCLoss and it gave reasonable results in no time. I am not able to understand why my approach did not work. If I find out in the future, I will share it here, otherwise; I think the CTCLoss works fine.