Sorry for the reply not being able to answer your error, but I wanted to ask you what method are you using for CTC decoding?
Tensorflow as options like CTC beam search decoder, or CTC greedy search decoder, have you tried to use TensorFlow method while using base PyTorch implementation.