Hi,
I tried to find a way around this problem but I am not sure why it did not work. I know that this is a “hack” and not recommended way.
I calculated the loss from network 1 which is present in device 1 with torch.no_grad(). Now the loss from network 1 (L1) and I am going to use it to backprop on Network 2 (same architecture) since the backward graph is not constructed I cannot directly do backprop on Network 2 and grad will be None. What I did was to generate a random tensor of the size required by Network 2 and some random numbers as labels. Then I ran forward prop with those randomly generated tensors and labels so that the backward graph is established. Now I calculate the loss of random label and model output (of course this does not make sense) but after the calculation of loss, I replace “data” (loss.data) with loss from network 1 assuming that the backprop will update weights based on replaced value.  I wanted to know if this should work technically or not.