Could the issue be related to this?  On pytorch’s CUDA Semantics 5 page, it says: “Unless you enable peer-to-peer memory access, any attempts to launch ops on tensors spread across different devices will raise an error.”
Does running MSELoss count as “launching ops”? I’d test it by enabling it, but there doesn’t seem to be any documentation on how to enable GPU peer-to-peer memory access on pytorch either here or on Google, apart from one post 5 where @smth indicates it’s on by default…