Solved by ptrblck in post #2 
              
              
                I’m not familiar with the usage in SRGAN, but in ResNets the identity used in the skip connections is not detached from the computation graph as seen e.g. here.