Hi,
The residual.requires_grad = True is not really useful here as you won’t be able to access the computed grad.
And the detach() will break the graph, that means that no gradient will flow back your residual connections. Not sure how that would impact the training, but you won’t have gradients anymore for sure.

RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True when calling backward the first time.

This can happen for a few reasons You will need to share some code.
Also does using out = out + residual solves your issue?