With the random label code you posted above, the one problem seems to be that 0 should not be in the labels. If I change the labels to
    labels = Variable(torch.from_numpy(np.random.randint(1, 10, (batch_size, 2))).int()).view(-1)

(note the lower limit of 1 in randint), I get losses of about 5±0.3.
The 0 label is reserved for blank rnn output with the warp_ctc implementation.
Best regards
Thomas