Autograd is able to create the backward pass through different devices.
The error in the first code was that y was passed to model2 (which was still on cuda:0), while z should be passed.
I’m wondering why the forward pass didn’t throw an error.
Anyway. after fixing this bug, the code should be working.