Solved by tom in post #2 
              
              
                Yes, zero initial hiddenstate is standard so much so that it is the default in nn.LSTM if you don’t pass in a hidden state (rather than, e.g. throwing an error).  Random initialization could also be used if zeros don’t work. Two basic ideas here: 

If your hidden state evolution is “ergodic”, the …