Sorry to bring up an old post, but related question: I’m also trying to learn the initial hidden state for an LSTM, but using a separate context network to do so (as in the paper “Multiple Object Recognition with Visual Attention”). I thought that the output of the context vector being a non-leaf Variable would have the gradients propagate back through the LSTM to the context network, but the gradients of the parameters in the context network are still None. What would be the correct way to go about doing this?