That is not how you’re supposed to use CTC loss. Do feed in the proper target lengths.
What happens is that the loss as you call it would require the model to output PAD BLANK PAD BLANK PAD … for as many pads as you have at the end (for repetitions, CTC needs two x elements to represent one y element) and so this cannot be represented in the x length. That in terns means the probability for the x representing y is 0 and the negative log probability (which is the loss) is inf.
Best regards
Thomas