I tried to train CTC model based on LibriSpeech dataset,
but CTCLoss always return inf or 0 if I use zero_infinity=True.
here is my decoder:
class ConvDecoder(nn.Module):
    def __init__(self, in_channels, vocab_size):
        super().__init__()
        self.decoder = nn.Conv1d(
            in_channels=in_channels, out_channels=vocab_size, kernel_size=1)

    def forward(self, x):
        return self.decoder(x)



It gets tensor of size (batch, channels, width) and return (batch, 28, width) where 28 is my vocab size, my vocab labels start from 1 and reserved label 0 for blank id.


Then I reshape model output to (width, batch, 28), here is exact code:


logits = self.forward(inputs)
batch_size, channels, sequence = logits.size()
logits = logits.view((sequence, batch_size, channels))
probs = nn.functional.log_softmax(logits, dim=-1)


And finally CTC loss:

loss = self.loss(probs, outputs, input_lengths, output_lengths)

how I instantiate loss:
self.loss = nn.CTCLoss(blank=0)
and this is my vocab dictionary:
{'a': 1,
 'b': 2,
 'c': 3,
 'd': 4,
 'e': 5,
 'f': 6,
 'g': 7,
 'h': 8,
 'i': 9,
 'j': 10,
 'k': 11,
 'l': 12,
 'm': 13,
 'n': 14,
 'o': 15,
 'p': 16,
 'q': 17,
 'r': 18,
 's': 19,
 't': 20,
 'u': 21,
 'v': 22,
 'w': 23,
 'x': 24,
 'y': 25,
 'z': 26,
 ' ': 27,
 '[unk]': 28}

indexes start from 1 but didn’t use any character for blank in my vocab.

I tired several examples of CTC models and they worked, just can’t figure out what I’ve done wrong.
here is train logs:
Epoch 0:   0%|          | 2/28539 [00:04<17:53:05,  2.26s/it, loss=inf, v_num=36]
output shape: torch.Size([732, 1, 28])

Epoch 0:   0%|          | 3/28539 [00:05<15:16:36,  1.93s/it, loss=inf, v_num=36]
output shape: torch.Size([1508, 1, 28])

Epoch 0:   0%|          | 4/28539 [00:08<16:28:45,  2.08s/it, loss=inf, v_num=36]
output shape: torch.Size([549, 1, 28])

note that output shape refers to final tensor that get into self.loss, set batch size to 1 for simplicity.
Thank you.