At the risk of being the CTC person that I don’t want to be:
The key that the paper is not using log space, so we are not interested in the derivative by y, but by log y. With this, we can remind ourselves that is that log probs are logits (i.e. x.log_softmax(dim=1) == x.log_softmax(dim=1).log_softmax(dim=1) ), so the derivative w.r.t. log y will be the one w.r.t. u.
The worst thing that happens is that you project the gradient onto the tangent space of the space (not a manifold, but close) of log probs twice (in CTC loss backward and then again in log_softmax backward). Is there a speedup to be had? In theory, yes, because the backward of the log_softmax you’ll be doing in PyTorch will compute the mean that we know is zero. But will we save from not doing it in the ctc backward? I haven’t thought about it much, but I expect not much, as we would need to specify ctc to take logits and take the logsoftmax in the ctc forward (but then we don’t need the backward).
Best regards
Thomas