Indeed, if we read this in isolation, it doesn’t make much sense.
If you don’t want to train the initial hidden state, you don’t need the gradient, if you want to train the initial hidden state as part of your model, it would make more sense to just have it as a parameter.
Best regards
Thomas