You are deleting the gradients after they were computed and before the weight updates were performed.
Try to move optimizer.zero_grad(), e.g.:
for epoch in range(epochs):
    for i in range(len(X)):
        optimizer.zero_grad()
        for ele in X[i]:
            output, hidden = rnn(Variable(ele.t()).cuda(), hidden.cuda())
        loss = criterion(output, Variable(Y[i]).cuda())
        
        loss.backward(retain_graph=True)
        optimizer.step()