It create the hidden state using input.data.new() which basically makes a new tensor of the same type and on the same device as input.data.
So if your LSTM input is on cuda, then the hidden state will be created on cuda too.