tom:

@ptrblck 's suggestion probably is the first important thing to look out for. If that is the cause, it means that for some of your data, the output length of your AlexNet is insufficient to be able to produce the target sequence.
If that doesn’t help in itself:

Be sure to use the latest PyTorch version.
If you want to find out what is going on,  you can do the following:

at the end of each training step (after optimizer.step()) can you check if any weight has a NaN,
look at the last inputs into CTC loss when a weight turned NaN (which means that the gradient was NaN).
does it happen for CPU as well?



Note that you need to check the weights for NaN, not the loss. Very likely the gradients  go NaN first and you’ll see the loss turn NaN in the next iter.
Things to look out for:

targets must not contain blank.
input length should be larger than target length + number of repeat positions (+1? I can’t remember).

Best regards
Thomas


sorry for the late reply, I have used zero_infinity = True but it didn’t solve the problem.
I found that when I use batch size is 1 there’s no problem, but when I use batch size more than 1 it always produces ‘NaN’ in weight and loss output when training passed few iterations.
so I think it possible that the problem occurs due to padding.
I padded the sequence of video input to have the length as the maximum input length in mini-batch with the last frame.
and I padded target to have the length as the maximum target length in mini-batch with ‘blank’.
I would like to ask you about my padding method is it the correct way ?.
the input length look like

[150 246 220 120]
(each element is the length of each input in batch, in this case, use batch size is 4)

the target look like

[[1 30 4 3 14 1 0 0 0 0]
[1 2 5 29 3 14 30 2 4 1]
[1 5 3 7 1 0 0 0 0 0]
[1 29 3 1 0 0 0 0 0 0 0]]
(padded 'blank ’ to have the length as the maximum target length in mini-batch)

and target length look like

[6 10 5 4]