Cheers, I’m really glad it helped! (and apologies to the mods and other readers for the cross post)
If each data instance is not related e.g, random shuffled, it is more appropriate to initialize the hidden state each time at the start of the sequence.


Beginning of every minibatch: This implies that we are using the hidden state from the previous sentence/sequence to initialise the next sequence. We might want to do this if the instances in the minibatch are related. Maybe a minibatch is a paragraph of sentences?


Beginning of every epoch: Same as above, except on a larger scale.


Values set to 0 or random: Both are valid approaches. There is less risk of overfitting if the hidden state is randomly initialized, as compared to 0 all the time. Having said that there are “stronger” ways of preventing overfitting like dropout which is tunable. So I would think if your goal is to squeeze the last bit of performance, try both. If you are implementing for research, just go with 0.


There are also more advanced methods like learning the initial hidden state, which is an ongoing area of research.