Having the input size as long as the target size is not a good idea.
If your target has repetitions, it won’t be representable and so the loss is infinite. If your target doesn’t have repetitions, there only is one valid alignment, and you might as well use per-sequence item cross entropy instead of CTC.
I would recommend staring a bit at the CTC article on Distill.pub 1 to get ideas about your modeling with special attention on the blank token aka ϵ. After reading it, take a sequence (using batch size 1 if your want) that produces inf loss and try to manually find a good alignment for CTC loss. Then you’ll have fully understood what is wrong and have achieved CTC-Zen.
Best regards
Thomas