Oh right. I forgot that the outputs also have both directions. Then your self.hidden2tag = nn.Linear(hidden_dim, tagset_size) should become self.hidden2tag = nn.Linear(hidden_dim * 2, tagset_size).  Hope that this works!