Arsham_mor:

batch_size, channels, sequence = logits.size()
logits = logits.view((sequence, batch_size, channels))



You almost certainly want permute here and not view.
A loss of inf means your input sequence is too short to be aligned to your target sequence (ie the data has likelihood 0 given the model - CTC loss is a negative log likelihood after all).
Best regards
Thomas