In your original model you needed to modify the initialisation of the hidden layers.
def __init__(self, embedding_dim, hidden_dim, hidden_layers, vocab_size, tagset_size):
    ...
    self.hidden_layers = hidden_layers
    ...

def init_hidden(self):
    return (autograd.Variable(torch.zeros(self.hidden_layers, 1, self.hidden_dim)),
        autograd.Variable(torch.zeros(self.hidden_layers, 1, self.hidden_dim)))

That should suffice for adding layers to the LSTM with your original code.
The RuntimeError: Trying to backward through the graph a second time happens because you keep the hidden state between batches, and you don’t detach or repackage the hidden state in between batches. What happens is that loss.backward() is trying to back-propagate all the way through to the start of time, which works for the first batch but not for the second because the graph for the first batch has been discarded.
There are two possible solutions.


detach/repackage the hidden state in between batches. There are (at least) three ways to do this.

self.hidden[0].detach_()
self.hidden[0] = self.hidden[0].detach()
self.hidden[0] = Variable(self.hidden[0].data, requires_grad=True)

and similarly for self.hidden[1]


replace loss.backward() with loss.backward(retain_graph=True) but know that each successive batch will take more time than the previous one because it will have to back-propagate all the way through to the start of the first batch.