Thabang_Lukhetho:

Why is the hidden state initialized to zero for each batch when doing a forward pass? And when are you supposed to this during training or testing or both?


You can do it for each epoch too (instead of each batch), which I think is better. You’ll need to do it for both training and testing (and validation).



 Thabang_Lukhetho:

If you initialized the hidden state with a batch_size=128 during training and for testing the batch_size=1, would I have to initialize the hidden state to 0 with a new batch size?


Yes, the initialization should match the batch_size of the input.



 Thabang_Lukhetho:

    hidden = (torch.zeros(self.layers, batch_size, self.hidden_size).to(device=device), torch.zeros(self.layers, batch_size, self.hidden_size).to(device=device))



hidden should be passed as the input the forward function, not initialized in the here. Check out an example here 19.