Everything computed inside forward will remain distributed among gpus. After returning forward’s output the batch is concatenated back in device 1.
The problem is that if you generate tensors inside the funcion, you have to prepare the code to make it device agnostic, such that every tensor you define inside the forward were properly allocated.
For example, inside forward, montecarlo, you are creating reward tensors initilized as zeros. That tensor is wrongly allocated in cpu. it requires something like torch.zeros().to(encoder_output.device)
In short, whatever coded inside forward is distributed