The error seems to be raised in:
rnn_layer(lstm_input, (tgt_hidden_states[layer_id], tgt_cell_states[layer_id]))

Could you check the .device attribute of all inputs as well as the parameters of rnn_layer and make sure they are on the GPU?
In case you get stuck, please post an executable code snippet, so that we could reproduce and debug the issue.