Solved by tom in post #2 
              
              
                For this you need to spell out the time loop using LSTMCell. That makes you pass the hidden state at each timestep and you can use hx.retain_grad() etc. 
Best regards 
Thomas