Yeah, just a quick experiment. It showed a very modest gain on the devset (less than 1% accuracy I think), and it seemed more prone to overfitting after reaching that peak.
I’ve read a couple things arguing in favour of this approach, but in terms of code it seems everyone uses zeros as h0 (pytorch’s default behaviour)