Can you set grads to True for net2’s parameters before you start the forward and then set them to false after you are done with the forward? This way the grads should be False for the entire backward pass despite concurrent execution of the backward pass across multiple GPUs.