Yes, you will have to move policy_output to the cpu yourself (pytorch never move stuff between the cpu and gpu automatically). But then yes the backprop will work just fine. Note that the q1 backprop will be computed on the cpu and the policy network one will be computed on the gpu in your particular case.