Out of curiosity, I trained a simple binary classifier (LSTM with attention) on a text dataset of mine. Once with random initialization, once with zero’d initialization of the hidden state for each batch:
ZEROES
Training accuracy: 0.970, test accuracy: 0.840
Training accuracy: 0.975, test accuracy: 0.841
Training accuracy: 0.970, test accuracy: 0.840

RANDOM
Training accuracy: 0.927, test accuracy: 0.847
Training accuracy: 0.923, test accuracy: 0.845
Training accuracy: 0.926, test accuracy: 0.849

The result are not unexpected, I think. Using zero’d hidden states yields a higher training accuracy since the same sentence never starts with a different hidden state. In fact, all sentences are treated equally given the initial hidden state is the same – I don’t think it’s important that the initial state is all zero’s, it’s just important that it’s the same for each batch (even if it’s set randomly at the very beginning).
The test accuracy is a tad better for a random initialization. I guess one could argue that a random initialization introduces some kind of regularization that avoids overfitting (lower training accuracy) but generalizes a bit better (higher test accuracy).
Disclaimer: This was just a quick-and-dirty test with a simple model and small-ish dataset. Please don’t use these results to make any deeper conclusions :).