Yes, Autograd will be able to calculate the gradients of the operations resulting in the first hidden state.
As long as you don’t detach the computation graph, e.g. by using tensor.data or tensor.detach(), you should be fine.
PS: Alternatively, slicing might be simpler, but I’m not sure if that’s suitable for your use case.