Hi Yan,
Make sure your src lengths are long enough in which src lengths >= target lengths or simply try using zero_infinity=True to avoid having infinite losses.
Other than that, training with CTC is a bit tricky, so expect to get a lot of blanks at early epochs as that is perfectly normal.
Hope that helps.