Instead of randomly (or setting 0) initializing the hidden state h0, I want the model to learn the RNN hidden state by itself. According to this article Non-Zero Initial States for Recurrent Neural Networks 416, learning the initial state can speed up training and improve generalization.
Following this post, I set the initial hidden state as a parameter in the module:
self.word_lstm_init_h = Parameter(torch.randn(2, 20, word_lstm_dim), requires_grad=True).type(FloatTensor)
self.word_lstm_init_c = Parameter(torch.randn(2, 20, word_lstm_dim), requires_grad=True).type(FloatTensor)
self.word_lstm = nn.LSTM(word_lstm_input_dim, word_lstm_dim, 1,
                                 bidirectional=True, batch_first=True)

For the forward function:
word_lstm_input = torch.nn.utils.rnn.pack_padded_sequence(
    word_lstm_input, seq_len, batch_first=True
)
word_lstm_out, word_lstm_h = self.word_lstm(
    word_lstm_input, (self.word_lstm_init_h, self.word_lstm_init_c)
)
word_lstm_out, _ = torch.nn.utils.rnn.pad_packed_sequence(
    word_lstm_out, batch_first=True
)

I’m skip the loss function here. After loss.backward(), I try to print out gradients of the initial state.
print(model.word_lstm_init_h.grad)
print(model.word_lstm_init_c.grad)

It gives me “None” for the both.
Does nn.LSTM pass gradients to the initial hidden state?