I do know that residual/skip connections can be implemented by simply doing
out = someOperation(x)
residual = x
out += residual
return out

but I am wondering if we have the same outcome by doing it in the following way
out = someOperation(x)

residual = x.detach().clone()
residual.requires_grad = True

out += residual
return out

Now, I know you’re asking yourself why would I even go into this trouble if the first thing works well, but, unfortunately, it doesn’t in my case. I believe it is because I am creating a partially invertible architecture and that the error might be caused by the implementation of the library for invertible architectures, but I am not completely sure since I do not understand it fully yet.
If my implementation follows the first case, I get
RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True when calling backward the first time.

The problem is not in performing backwards() separately for losses or anything such, so that is why I assume it may be caused by the invertible architectures library.
However, by implementing it as shown in the second case (stumbled upon it here 1), it does train, but I am worried if the gradients are kept the way they should be, which is why I’m looking for the answer here. Thanks!