The hidden state will certainly be overwritten at each state. But you have to initialize an h_0 at the first state. Normally for each iteration, h_0 is randomly initialized. In here, I tried to set h_0 as a parameter and let the model learns an h_0 so that at each iteration, h_0 is fixed to a leaned value. The hypothesis is that the learned h_0 can either improve performance or speed up the training process.