I can’t see the model learning the initial state. init_hidden() gets called for every call of the forward() method, i.e., for each batch. So each batch starts with a new random initial state. What exactly is learned here?
As far as I can tell, the learning the initial state is done by initializing the hidden state once when creating the model, and then you only detach() the hidden state for each new batch.
Creating a new random hidden state for each batch probably doesn’t hurt much – I don’t know, to be honest. But it certainly results in the case where the same example – input sequence and target sequence/class) is trained with different initial initial hidden states. Not sure which effects this has.