I have updated the post with error that I got. I tried your approach which was similar to mine except that I did not use P2P GPU transfer. I still get the same error. RuntimeError: tensors are on different GPUs