mebrunet:

I therefore assume higher order gradients (e.g. HVPs) through CTCLoss won’t work.


Indeed. I’d recommend starting from the original paper 7, you could differentiate equations 15 and the derivative below it to compute the second derivative. I’ve tried to comment the source to make it easy to follow with the paper in mind (but remember that alpha and beta are computed in log space).
I’m not sure whether it will work well numerically - even for the first derivative of ctc_loss, the numerical precision can be touchy.
I’d be interested to see your results.
Best regards
Thomas