Thank you for your reply, I have transposed all int tensor to long. I also place log_probs & targets to cuda, place lengths to cpu. But I found that the speed of placing log_probs & targets on cuda or cpu are nearly same. Why?  should I always use cpu?
def pytorch_ctc_loss(pre, gt, gtlen):
    """
    Input:
        pre   : (T,B,C), FloatTensor
        gt    : (sum(target_lengths)), IntTensor
        gtlen : B, IntTensor
    """
    pre = F.log_softmax(pre, dim=-1)
    prlen = torch.LongTensor([pre.size(0)]*pre.size(1))
    return F.ctc_loss(pre, gt.long().to(pre.device), prlen, gtlen.long().cpu(), blank=0)