UPDATE:
I figured out what was wrong w/ the hidden state… I had two hidden states. I should have had only one since it’s a GRU.
The training error still increases during training so still looking for a answer for that.
self.hidden_cell = torch.zeros(1, 1, self.hidden_layer_size)

model.hidden_cell = torch.zeros(1, 1, model.hidden_layer_size)