Hi Ptr,
I have updated the code. Let’s assume that I am only interested in performing the following 3. steps (I agree it seems bit unusual NN).
In the below, I will explain in summary the key aspects of what I am trying to do:
Step 1. calculate a scalar quantity called scalar in step 1.
Step 2. backpopagate to update net.
Step 3… modify parameters of other_net based on some function of scalar and loss
Now my problem is that each GPU computes different value of scalar. I want to average this scalar over all the 8 GPUs. The code doesn’t work and returns error “tensors on different GPUs”. So I explicitly start 8 processes, and each NN have a self.rank = rank (rank of process), and each variable etc. have .cuda(self.rank) at the end. However, GPU utilization becomes very bad.