You should not pad target sequences with blank, that’s the main reason your model predicts blanks.
In general is harder to train end-2-end the full architecture only with CTC Loss depending on the model architecture and depth
Also try to use reduction=‘mean’ and zero_infinity=“True” , I found it works better with these parameters.