I guess it might come due to feeding the hidden output back to the model during the epoch, while you are reinitializing it for each new epoch.
How were you planning on handling the hidden state during testing?