Ah, sorry. Somehow I’ve missed these lines.
Your code should work, if you assign the tensors back:
initial_hidden_state = initial_hidden_state.cuda()
initial_cell_state = initial_cell_state.cuda()

You need this assignment for tensors, while modules will be moved “in-place”.
Which line is your new error pointing to?