I might be wrong, but I don’t think that’s easily doable at the moment, since the forward pass on the first device would also create all intermediate tensors on this device.
While you could move the complete model to a different device, the computation graph would still be on the first device.
Maybe there are some 3rd party libs or I’m missing something, so let’s wait for some experts to chime in.